[jira] [Created] (HBASE-16690) Move znode path configs to a separated class

[Duo Zhang (JIRA)]

Duo Zhang created HBASE-16690:
-

 Summary: Move znode path configs to a separated class
 Key: HBASE-16690
 URL: https://issues.apache.org/jira/browse/HBASE-16690
 Project: HBase
  Issue Type: Improvement
Reporter: Duo Zhang


Which makes it easier to use curator at client in the future.

And also, will try to fix the bad practise that declares masterMaintZNode as 
static but update it a non-static method.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16691) optimize KeyOnlyFilter

[binlijin (JIRA)]

binlijin created HBASE-16691:


 Summary: optimize KeyOnlyFilter
 Key: HBASE-16691
 URL: https://issues.apache.org/jira/browse/HBASE-16691
 Project: HBase
  Issue Type: Improvement
Reporter: binlijin
Priority: Minor


In KeyOnlyFilter#transformCell will return a KeyOnlyCell that have no value or 
has valueLength as value, current will copy all row keys into a new byte[] and 
new a KeyValue, we can eliminate the copy and have a wrap KeyOnlyCell that 
ignore the cell's value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16692) Make ByteBufferUtils#equals more safe and correct

[binlijin (JIRA)]

binlijin created HBASE-16692:


 Summary: Make ByteBufferUtils#equals more safe and correct
 Key: HBASE-16692
 URL: https://issues.apache.org/jira/browse/HBASE-16692
 Project: HBase
  Issue Type: Improvement
Reporter: binlijin


ByteBufferUtils.equals(HConstants.EMPTY_BYTE_BUFFER, 0, 0, 
HConstants.EMPTY_BYTE_ARRAY, 0, 0) will throw 
java.lang.ArrayIndexOutOfBoundsException: -1, i think it should return true.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16693) The commit of HBASE-16604 creates a unrelated folder

[ChiaPing Tsai (JIRA)]

ChiaPing Tsai created HBASE-16693:
-

 Summary: The commit of HBASE-16604 creates a unrelated folder
 Key: HBASE-16693
 URL: https://issues.apache.org/jira/browse/HBASE-16693
 Project: HBase
  Issue Type: Bug
Reporter: ChiaPing Tsai
Priority: Critical


The folder's name is "hbase-thrift\036src", and it causes the git-clone failure 
on windows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSSION] MR jobs started by Master or RS

[Ted Yu]

I read through HADOOP-13433
 - the cited race
condition is in jdk.

Suggest pinging the reviewer on JIRA to get it moving.

bq. But the ugly code in HMaster is readlly a problem...

Can you be specific as to which code is ugly ? Is it in the backup /
restore mega patch ?

Cheers

On Thu, Sep 22, 2016 at 10:44 PM, 张铎  wrote:

> If you guys have already implemented the feature in the MR way and the
> patch is ready for landing on master, I'm a -0 on it as I do not want to
> block the development progress.
>
> But I strongly suggest later we need to revisit the design and see if we
> can seperated the logic from HMaster as much as possible. HA is not a big
> problem if you do not store any metada locally. But the ugly code in
> HMaster is readlly a problem...
>
> And for security, I have a issue pending for a long time. Can someone help
> taking a simple look at it? This is what I mean, ugly code... logout and
> destroy the credentials in a subject when it is still being used, and
> declared as LimitPrivacy so I can not change the behivor and the only way
> to fix it is to write another piece of ugly code...
>
> https://issues.apache.org/jira/browse/HADOOP-13433
>
> 2016-09-23 12:53 GMT+08:00 Vladimir Rodionov :
>
> > >> If in the future, we find better ways of doing this without using MR,
> we
> > can certainly consider that
> >
> > Our framework for distributed operations is abstract and allows
> > different implementations. MR is just one implementation we provide.
> >
> > -Vlad
> >
> > On Thu, Sep 22, 2016 at 9:38 PM, Devaraj Das 
> wrote:
> >
> > > Guys, first off apologies for bringing in the topic of MR-based
> > > compactions.. But I was thinking more about the SpliceMachine approach
> of
> > > managing compactions in Spark where apparently they saw a lot of
> > benefits.
> > > Apologies for giving you that sore throat Andrew; I really didn't mean
> to
> > > :-)
> > >
> > > So on this issue, we have these on the plate:
> > > 0. Somehow not use MR but something like that
> > > 1. Run a standalone service other than master
> > > 2. Shell out from the master
> > >
> > > I don't think we have a good answer to (0), and I don't think it's even
> > > worth the effort of trying to build something when MR is already there,
> > and
> > > being used by HBase already for some operations.
> > >
> > > On (1), we have to deal with a myriad of issues - HA of the server not
> > > being the least of them all. Security (kerberos authentication, another
> > > keytab to manage, etc. etc. etc.). IMO, that approach is DOA. Instead
> > let's
> > > substitute that (1) with the HBase Master. I haven't seen any good
> reason
> > > why the HBase master shouldn't launch MR jobs if needed. It's not
> ideal;
> > > agreed.
> > >
> > > Now before going to (2), let's see what are the benefits of running the
> > > backup/restore jobs from the master. I think Ted has summarized some of
> > the
> > > issues that we need to take care of - basically, the master can keep
> > track
> > > of running jobs, and should it fail, the backup master can continue
> > keeping
> > > track of it (since the jobId would have been recorded in the proc WAL).
> > The
> > > master can also do cleanup, etc. of failed backup/restore processes.
> > > Security is another issue - the job needs to run as 'hbase' since it
> owns
> > > the data. Having the master launch the job makes it get that privilege.
> > In
> > > the (2) approach, it's hard to do some of the above management.
> > >
> > > Guys, just to reiterate, the patch as such is ready from the overall
> > > design/arch point of view (maybe code review is still pending from
> > Matteo).
> > > If in the future, we find better ways of doing this without using MR,
> we
> > > can certainly consider that. But IMO don't think we should block this
> > patch
> > > from getting merged.
> > >
> > > 
> > > From: 张铎 
> > > Sent: Thursday, September 22, 2016 8:32 PM
> > > To: dev@hbase.apache.org
> > > Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> > >
> > > So what about a standalone service other than master? You can use your
> > own
> > > procedure store in that service?
> > >
> > > 2016-09-23 11:28 GMT+08:00 Ted Yu :
> > >
> > > > An earlier implementation was client driven.
> > > >
> > > > But with that approach, it is hard to resume if there is error
> midway.
> > > > Using Procedure V2 makes the backup / restore more robust.
> > > >
> > > > Another consideration is for security. It is hard to enforce security
> > (to
> > > > be implemented) for client driven actions.
> > > >
> > > > Cheers
> > > >
> > > > > On Sep 22, 2016, at 8:15 PM, Andrew Purtell <
> > andrew.purt...@gmail.com>
> > > > wrote:
> > > > >
> > > > > No, this misses Matteo's finer point, which is "shelling out" from
> > the
> > > > master directly to run MR is a first. Why not drive this with a
> utility
> > > > derived from Tool?
> > > > >
> > 

Re: [DISCUSSION] MR jobs started by Master or RS

[张铎]

No, not your fault, at lease, not this time:)

Why I call the code ugly? Can you simply tell me the sequence of calls when
starting up the HMaster? HMaster is also a regionserver so it extends
HRegionServer, and the initialization of HRegionServer sometimes needs to
make rpc calls to HMaster. A simple change would cause probabilistic dead
lock or some strange NPEs...

That's why I'm very nervous when somebody wants to add new features or add
external dependencies to HMaster, especially add more works for the start
up processing...

Thanks.

2016-09-23 20:02 GMT+08:00 Ted Yu :

> I read through HADOOP-13433
>  - the cited race
> condition is in jdk.
>
> Suggest pinging the reviewer on JIRA to get it moving.
>
> bq. But the ugly code in HMaster is readlly a problem...
>
> Can you be specific as to which code is ugly ? Is it in the backup /
> restore mega patch ?
>
> Cheers
>
> On Thu, Sep 22, 2016 at 10:44 PM, 张铎  wrote:
>
> > If you guys have already implemented the feature in the MR way and the
> > patch is ready for landing on master, I'm a -0 on it as I do not want to
> > block the development progress.
> >
> > But I strongly suggest later we need to revisit the design and see if we
> > can seperated the logic from HMaster as much as possible. HA is not a big
> > problem if you do not store any metada locally. But the ugly code in
> > HMaster is readlly a problem...
> >
> > And for security, I have a issue pending for a long time. Can someone
> help
> > taking a simple look at it? This is what I mean, ugly code... logout and
> > destroy the credentials in a subject when it is still being used, and
> > declared as LimitPrivacy so I can not change the behivor and the only way
> > to fix it is to write another piece of ugly code...
> >
> > https://issues.apache.org/jira/browse/HADOOP-13433
> >
> > 2016-09-23 12:53 GMT+08:00 Vladimir Rodionov :
> >
> > > >> If in the future, we find better ways of doing this without using
> MR,
> > we
> > > can certainly consider that
> > >
> > > Our framework for distributed operations is abstract and allows
> > > different implementations. MR is just one implementation we provide.
> > >
> > > -Vlad
> > >
> > > On Thu, Sep 22, 2016 at 9:38 PM, Devaraj Das 
> > wrote:
> > >
> > > > Guys, first off apologies for bringing in the topic of MR-based
> > > > compactions.. But I was thinking more about the SpliceMachine
> approach
> > of
> > > > managing compactions in Spark where apparently they saw a lot of
> > > benefits.
> > > > Apologies for giving you that sore throat Andrew; I really didn't
> mean
> > to
> > > > :-)
> > > >
> > > > So on this issue, we have these on the plate:
> > > > 0. Somehow not use MR but something like that
> > > > 1. Run a standalone service other than master
> > > > 2. Shell out from the master
> > > >
> > > > I don't think we have a good answer to (0), and I don't think it's
> even
> > > > worth the effort of trying to build something when MR is already
> there,
> > > and
> > > > being used by HBase already for some operations.
> > > >
> > > > On (1), we have to deal with a myriad of issues - HA of the server
> not
> > > > being the least of them all. Security (kerberos authentication,
> another
> > > > keytab to manage, etc. etc. etc.). IMO, that approach is DOA. Instead
> > > let's
> > > > substitute that (1) with the HBase Master. I haven't seen any good
> > reason
> > > > why the HBase master shouldn't launch MR jobs if needed. It's not
> > ideal;
> > > > agreed.
> > > >
> > > > Now before going to (2), let's see what are the benefits of running
> the
> > > > backup/restore jobs from the master. I think Ted has summarized some
> of
> > > the
> > > > issues that we need to take care of - basically, the master can keep
> > > track
> > > > of running jobs, and should it fail, the backup master can continue
> > > keeping
> > > > track of it (since the jobId would have been recorded in the proc
> WAL).
> > > The
> > > > master can also do cleanup, etc. of failed backup/restore processes.
> > > > Security is another issue - the job needs to run as 'hbase' since it
> > owns
> > > > the data. Having the master launch the job makes it get that
> privilege.
> > > In
> > > > the (2) approach, it's hard to do some of the above management.
> > > >
> > > > Guys, just to reiterate, the patch as such is ready from the overall
> > > > design/arch point of view (maybe code review is still pending from
> > > Matteo).
> > > > If in the future, we find better ways of doing this without using MR,
> > we
> > > > can certainly consider that. But IMO don't think we should block this
> > > patch
> > > > from getting merged.
> > > >
> > > > 
> > > > From: 张铎 
> > > > Sent: Thursday, September 22, 2016 8:32 PM
> > > > To: dev@hbase.apache.org
> > > > Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> > > >
> > > > So what about a standalone service other than master? You ca

Re: [DISCUSSION] MR jobs started by Master or RS

[Ted Yu]

I suggest you look at Matteo's work for AssignmentManager which is to make
Master more stable.

Cheers

On Fri, Sep 23, 2016 at 5:32 AM, 张铎  wrote:

> No, not your fault, at lease, not this time:)
>
> Why I call the code ugly? Can you simply tell me the sequence of calls when
> starting up the HMaster? HMaster is also a regionserver so it extends
> HRegionServer, and the initialization of HRegionServer sometimes needs to
> make rpc calls to HMaster. A simple change would cause probabilistic dead
> lock or some strange NPEs...
>
> That's why I'm very nervous when somebody wants to add new features or add
> external dependencies to HMaster, especially add more works for the start
> up processing...
>
> Thanks.
>
> 2016-09-23 20:02 GMT+08:00 Ted Yu :
>
> > I read through HADOOP-13433
> >  - the cited race
> > condition is in jdk.
> >
> > Suggest pinging the reviewer on JIRA to get it moving.
> >
> > bq. But the ugly code in HMaster is readlly a problem...
> >
> > Can you be specific as to which code is ugly ? Is it in the backup /
> > restore mega patch ?
> >
> > Cheers
> >
> > On Thu, Sep 22, 2016 at 10:44 PM, 张铎  wrote:
> >
> > > If you guys have already implemented the feature in the MR way and the
> > > patch is ready for landing on master, I'm a -0 on it as I do not want
> to
> > > block the development progress.
> > >
> > > But I strongly suggest later we need to revisit the design and see if
> we
> > > can seperated the logic from HMaster as much as possible. HA is not a
> big
> > > problem if you do not store any metada locally. But the ugly code in
> > > HMaster is readlly a problem...
> > >
> > > And for security, I have a issue pending for a long time. Can someone
> > help
> > > taking a simple look at it? This is what I mean, ugly code... logout
> and
> > > destroy the credentials in a subject when it is still being used, and
> > > declared as LimitPrivacy so I can not change the behivor and the only
> way
> > > to fix it is to write another piece of ugly code...
> > >
> > > https://issues.apache.org/jira/browse/HADOOP-13433
> > >
> > > 2016-09-23 12:53 GMT+08:00 Vladimir Rodionov :
> > >
> > > > >> If in the future, we find better ways of doing this without using
> > MR,
> > > we
> > > > can certainly consider that
> > > >
> > > > Our framework for distributed operations is abstract and allows
> > > > different implementations. MR is just one implementation we provide.
> > > >
> > > > -Vlad
> > > >
> > > > On Thu, Sep 22, 2016 at 9:38 PM, Devaraj Das 
> > > wrote:
> > > >
> > > > > Guys, first off apologies for bringing in the topic of MR-based
> > > > > compactions.. But I was thinking more about the SpliceMachine
> > approach
> > > of
> > > > > managing compactions in Spark where apparently they saw a lot of
> > > > benefits.
> > > > > Apologies for giving you that sore throat Andrew; I really didn't
> > mean
> > > to
> > > > > :-)
> > > > >
> > > > > So on this issue, we have these on the plate:
> > > > > 0. Somehow not use MR but something like that
> > > > > 1. Run a standalone service other than master
> > > > > 2. Shell out from the master
> > > > >
> > > > > I don't think we have a good answer to (0), and I don't think it's
> > even
> > > > > worth the effort of trying to build something when MR is already
> > there,
> > > > and
> > > > > being used by HBase already for some operations.
> > > > >
> > > > > On (1), we have to deal with a myriad of issues - HA of the server
> > not
> > > > > being the least of them all. Security (kerberos authentication,
> > another
> > > > > keytab to manage, etc. etc. etc.). IMO, that approach is DOA.
> Instead
> > > > let's
> > > > > substitute that (1) with the HBase Master. I haven't seen any good
> > > reason
> > > > > why the HBase master shouldn't launch MR jobs if needed. It's not
> > > ideal;
> > > > > agreed.
> > > > >
> > > > > Now before going to (2), let's see what are the benefits of running
> > the
> > > > > backup/restore jobs from the master. I think Ted has summarized
> some
> > of
> > > > the
> > > > > issues that we need to take care of - basically, the master can
> keep
> > > > track
> > > > > of running jobs, and should it fail, the backup master can continue
> > > > keeping
> > > > > track of it (since the jobId would have been recorded in the proc
> > WAL).
> > > > The
> > > > > master can also do cleanup, etc. of failed backup/restore
> processes.
> > > > > Security is another issue - the job needs to run as 'hbase' since
> it
> > > owns
> > > > > the data. Having the master launch the job makes it get that
> > privilege.
> > > > In
> > > > > the (2) approach, it's hard to do some of the above management.
> > > > >
> > > > > Guys, just to reiterate, the patch as such is ready from the
> overall
> > > > > design/arch point of view (maybe code review is still pending from
> > > > Matteo).
> > > > > If in the future, we find better ways of doing this without using
> MR,
> > > we
> > 

[jira] [Created] (HBASE-16694) Reduce garbage for onDiskChecksum in HFileBlock

[binlijin (JIRA)]

binlijin created HBASE-16694:


 Summary: Reduce garbage for onDiskChecksum in HFileBlock
 Key: HBASE-16694
 URL: https://issues.apache.org/jira/browse/HBASE-16694
 Project: HBase
  Issue Type: Improvement
Affects Versions: 2.0..
Reporter: binlijin
Assignee: binlijin
Priority: Minor


Current when finish a HFileBlock will create a new byte[] for onDiskChecksum, 
we can reuse it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSSION] MR jobs started by Master or RS

[Matteo Bertozzi]

let me try to go back to my original topic.
this question was meant to be generic, and provide some rule for future
code.

from what I can gather, a rule that may satisfy everyone can be:
 - we don't want any core feature (e.g. compaction/log-split/log-reply)
over MR, because some cluster may not want or may have an
external/uncontrolled MR setup.
 - we allow non-core features (e.g. features enabled by a flag) to run MR
jobs from hbase, because unless you use the feature, MR is not required.

Matteo


On Fri, Sep 23, 2016 at 5:39 AM, Ted Yu  wrote:

> I suggest you look at Matteo's work for AssignmentManager which is to make
> Master more stable.
>
> Cheers
>
> On Fri, Sep 23, 2016 at 5:32 AM, 张铎  wrote:
>
> > No, not your fault, at lease, not this time:)
> >
> > Why I call the code ugly? Can you simply tell me the sequence of calls
> when
> > starting up the HMaster? HMaster is also a regionserver so it extends
> > HRegionServer, and the initialization of HRegionServer sometimes needs to
> > make rpc calls to HMaster. A simple change would cause probabilistic dead
> > lock or some strange NPEs...
> >
> > That's why I'm very nervous when somebody wants to add new features or
> add
> > external dependencies to HMaster, especially add more works for the start
> > up processing...
> >
> > Thanks.
> >
> > 2016-09-23 20:02 GMT+08:00 Ted Yu :
> >
> > > I read through HADOOP-13433
> > >  - the cited race
> > > condition is in jdk.
> > >
> > > Suggest pinging the reviewer on JIRA to get it moving.
> > >
> > > bq. But the ugly code in HMaster is readlly a problem...
> > >
> > > Can you be specific as to which code is ugly ? Is it in the backup /
> > > restore mega patch ?
> > >
> > > Cheers
> > >
> > > On Thu, Sep 22, 2016 at 10:44 PM, 张铎  wrote:
> > >
> > > > If you guys have already implemented the feature in the MR way and
> the
> > > > patch is ready for landing on master, I'm a -0 on it as I do not want
> > to
> > > > block the development progress.
> > > >
> > > > But I strongly suggest later we need to revisit the design and see if
> > we
> > > > can seperated the logic from HMaster as much as possible. HA is not a
> > big
> > > > problem if you do not store any metada locally. But the ugly code in
> > > > HMaster is readlly a problem...
> > > >
> > > > And for security, I have a issue pending for a long time. Can someone
> > > help
> > > > taking a simple look at it? This is what I mean, ugly code... logout
> > and
> > > > destroy the credentials in a subject when it is still being used, and
> > > > declared as LimitPrivacy so I can not change the behivor and the only
> > way
> > > > to fix it is to write another piece of ugly code...
> > > >
> > > > https://issues.apache.org/jira/browse/HADOOP-13433
> > > >
> > > > 2016-09-23 12:53 GMT+08:00 Vladimir Rodionov  >:
> > > >
> > > > > >> If in the future, we find better ways of doing this without
> using
> > > MR,
> > > > we
> > > > > can certainly consider that
> > > > >
> > > > > Our framework for distributed operations is abstract and allows
> > > > > different implementations. MR is just one implementation we
> provide.
> > > > >
> > > > > -Vlad
> > > > >
> > > > > On Thu, Sep 22, 2016 at 9:38 PM, Devaraj Das  >
> > > > wrote:
> > > > >
> > > > > > Guys, first off apologies for bringing in the topic of MR-based
> > > > > > compactions.. But I was thinking more about the SpliceMachine
> > > approach
> > > > of
> > > > > > managing compactions in Spark where apparently they saw a lot of
> > > > > benefits.
> > > > > > Apologies for giving you that sore throat Andrew; I really didn't
> > > mean
> > > > to
> > > > > > :-)
> > > > > >
> > > > > > So on this issue, we have these on the plate:
> > > > > > 0. Somehow not use MR but something like that
> > > > > > 1. Run a standalone service other than master
> > > > > > 2. Shell out from the master
> > > > > >
> > > > > > I don't think we have a good answer to (0), and I don't think
> it's
> > > even
> > > > > > worth the effort of trying to build something when MR is already
> > > there,
> > > > > and
> > > > > > being used by HBase already for some operations.
> > > > > >
> > > > > > On (1), we have to deal with a myriad of issues - HA of the
> server
> > > not
> > > > > > being the least of them all. Security (kerberos authentication,
> > > another
> > > > > > keytab to manage, etc. etc. etc.). IMO, that approach is DOA.
> > Instead
> > > > > let's
> > > > > > substitute that (1) with the HBase Master. I haven't seen any
> good
> > > > reason
> > > > > > why the HBase master shouldn't launch MR jobs if needed. It's not
> > > > ideal;
> > > > > > agreed.
> > > > > >
> > > > > > Now before going to (2), let's see what are the benefits of
> running
> > > the
> > > > > > backup/restore jobs from the master. I think Ted has summarized
> > some
> > > of
> > > > > the
> > > > > > issues that we need to take care of - basically, the master can
> > keep
> > > > > track

[jira] [Created] (HBASE-16695) Procedure v2 - Support for parent holding locks

[Matteo Bertozzi (JIRA)]

Matteo Bertozzi created HBASE-16695:
---

 Summary: Procedure v2 - Support for parent holding locks
 Key: HBASE-16695
 URL: https://issues.apache.org/jira/browse/HBASE-16695
 Project: HBase
  Issue Type: Sub-task
  Components: proc-v2
Affects Versions: 2.0.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
 Fix For: 2.0.0


Add the logic to allow child procs to be executed when the parent is holding 
the xlock. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16696) TestBlockEvictionFromClient fails in master branch

[Ted Yu (JIRA)]

Ted Yu created HBASE-16696:
--

 Summary: TestBlockEvictionFromClient fails in master branch
 Key: HBASE-16696
 URL: https://issues.apache.org/jira/browse/HBASE-16696
 Project: HBase
  Issue Type: Bug
Reporter: Ted Yu
 Attachments: build-1638.out, build-1639.out

TestBlockEvictionFromClient consistently fails in master branch.

>From existing Jenkins builds, looks like this started with build 1639.

See attached Jenkins console logs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Successful: HBase Generate Website

[Apache Jenkins Server]

Build status: Successful

If successful, the website and docs have been generated. To update the live 
site, follow the instructions below. If failed, skip to the bottom of this 
email.

Use the following commands to download the patch and apply it to a clean branch 
based on origin/asf-site. If you prefer to keep the hbase-site repo around 
permanently, you can skip the clone step.

  git clone https://git-wip-us.apache.org/repos/asf/hbase-site.git

  cd hbase-site
  wget -O- 
https://builds.apache.org/job/hbase_generate_website/353/artifact/website.patch.zip
 | funzip > 946c1ed8f89967b1f036ee3b0dcc296082eee487.patch
  git fetch
  git checkout -b asf-site-946c1ed8f89967b1f036ee3b0dcc296082eee487 
origin/asf-site
  git am --whitespace=fix 946c1ed8f89967b1f036ee3b0dcc296082eee487.patch

At this point, you can preview the changes by opening index.html or any of the 
other HTML pages in your local 
asf-site-946c1ed8f89967b1f036ee3b0dcc296082eee487 branch.

There are lots of spurious changes, such as timestamps and CSS styles in 
tables, so a generic git diff is not very useful. To see a list of files that 
have been added, deleted, renamed, changed type, or are otherwise interesting, 
use the following command:

  git diff --name-status --diff-filter=ADCRTXUB origin/asf-site

To see only files that had 100 or more lines changed:

  git diff --stat origin/asf-site | grep -E '[1-9][0-9]{2,}'

When you are satisfied, publish your changes to origin/asf-site using these 
commands:

  git commit --allow-empty -m "Empty commit" # to work around a current ASF 
INFRA bug
  git push origin asf-site-946c1ed8f89967b1f036ee3b0dcc296082eee487:asf-site
  git checkout asf-site
  git branch -D asf-site-946c1ed8f89967b1f036ee3b0dcc296082eee487

Changes take a couple of minutes to be propagated. You can verify whether they 
have been propagated by looking at the Last Published date at the bottom of 
http://hbase.apache.org/. It should match the date in the index.html on the 
asf-site branch in Git.

As a courtesy- reply-all to this email to let other committers know you pushed 
the site.



If failed, see https://builds.apache.org/job/hbase_generate_website/353/console

Re: [DISCUSSION] MR jobs started by Master or RS

[Nick Dimiduk]

So we support MR/YARN anyway through these optional features, thus
burdening is with a baseline of maintenance. Since we have that anyway, do
we accept _optional_ implementations of "core" features based on MR it even
YARN? I'd guess HBase is co-deployed with a YARN or Spark cluster often
enough that shipping alternative implementations that can take advantage of
this infrastructure is quite reasonable -- even thoughtful!

The trouble with multiple implementations is the risk of one path or
another left to languish. We have a means to deal with this already, which
is to say if there's a champion up for maintaining it, and it makes good
sense, we accept it.

On Friday, September 23, 2016, Matteo Bertozzi 
wrote:

> let me try to go back to my original topic.
> this question was meant to be generic, and provide some rule for future
> code.
>
> from what I can gather, a rule that may satisfy everyone can be:
>  - we don't want any core feature (e.g. compaction/log-split/log-reply)
> over MR, because some cluster may not want or may have an
> external/uncontrolled MR setup.
>  - we allow non-core features (e.g. features enabled by a flag) to run MR
> jobs from hbase, because unless you use the feature, MR is not required.
>
> Matteo
>
>
> On Fri, Sep 23, 2016 at 5:39 AM, Ted Yu  > wrote:
>
> > I suggest you look at Matteo's work for AssignmentManager which is to
> make
> > Master more stable.
> >
> > Cheers
> >
> > On Fri, Sep 23, 2016 at 5:32 AM, 张铎  > wrote:
> >
> > > No, not your fault, at lease, not this time:)
> > >
> > > Why I call the code ugly? Can you simply tell me the sequence of calls
> > when
> > > starting up the HMaster? HMaster is also a regionserver so it extends
> > > HRegionServer, and the initialization of HRegionServer sometimes needs
> to
> > > make rpc calls to HMaster. A simple change would cause probabilistic
> dead
> > > lock or some strange NPEs...
> > >
> > > That's why I'm very nervous when somebody wants to add new features or
> > add
> > > external dependencies to HMaster, especially add more works for the
> start
> > > up processing...
> > >
> > > Thanks.
> > >
> > > 2016-09-23 20:02 GMT+08:00 Ted Yu 
> >:
> > >
> > > > I read through HADOOP-13433
> > > >  - the cited
> race
> > > > condition is in jdk.
> > > >
> > > > Suggest pinging the reviewer on JIRA to get it moving.
> > > >
> > > > bq. But the ugly code in HMaster is readlly a problem...
> > > >
> > > > Can you be specific as to which code is ugly ? Is it in the backup /
> > > > restore mega patch ?
> > > >
> > > > Cheers
> > > >
> > > > On Thu, Sep 22, 2016 at 10:44 PM, 张铎  > wrote:
> > > >
> > > > > If you guys have already implemented the feature in the MR way and
> > the
> > > > > patch is ready for landing on master, I'm a -0 on it as I do not
> want
> > > to
> > > > > block the development progress.
> > > > >
> > > > > But I strongly suggest later we need to revisit the design and see
> if
> > > we
> > > > > can seperated the logic from HMaster as much as possible. HA is
> not a
> > > big
> > > > > problem if you do not store any metada locally. But the ugly code
> in
> > > > > HMaster is readlly a problem...
> > > > >
> > > > > And for security, I have a issue pending for a long time. Can
> someone
> > > > help
> > > > > taking a simple look at it? This is what I mean, ugly code...
> logout
> > > and
> > > > > destroy the credentials in a subject when it is still being used,
> and
> > > > > declared as LimitPrivacy so I can not change the behivor and the
> only
> > > way
> > > > > to fix it is to write another piece of ugly code...
> > > > >
> > > > > https://issues.apache.org/jira/browse/HADOOP-13433
> > > > >
> > > > > 2016-09-23 12:53 GMT+08:00 Vladimir Rodionov <
> vladrodio...@gmail.com 
> > >:
> > > > >
> > > > > > >> If in the future, we find better ways of doing this without
> > using
> > > > MR,
> > > > > we
> > > > > > can certainly consider that
> > > > > >
> > > > > > Our framework for distributed operations is abstract and allows
> > > > > > different implementations. MR is just one implementation we
> > provide.
> > > > > >
> > > > > > -Vlad
> > > > > >
> > > > > > On Thu, Sep 22, 2016 at 9:38 PM, Devaraj Das <
> d...@hortonworks.com 
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Guys, first off apologies for bringing in the topic of MR-based
> > > > > > > compactions.. But I was thinking more about the SpliceMachine
> > > > approach
> > > > > of
> > > > > > > managing compactions in Spark where apparently they saw a lot
> of
> > > > > > benefits.
> > > > > > > Apologies for giving you that sore throat Andrew; I really
> didn't
> > > > mean
> > > > > to
> > > > > > > :-)
> > > > > > >
> > > > > > > So on this issue, we have these on the plate:
> > > > > > > 0. Somehow not use MR but something like that
> > > > > > > 1. Run a standalone service other than master
> > > > > > > 2. Shell out from the master
> > > > > > >
> > > > > > > I don't think we

Re: [DISCUSSION] MR jobs started by Master or RS

[Jerry He]

That is the point, Matteo.

Put it another way, there is nothing that stops a user from deploying
custom procedure, custom co-processor that calls out MR job.
The optional feature should satisfy some basic requirements. .e.g. No
impact if not deployed or used.  Limited impact if used.
It can be made with isolated dynamic loading of extra configuration (Yarn),
non-blocking non-occupying on the server handlers, or separate handler.
The impact would mostly be on the overall cluster resources. In this sense,
there is no difference, using another standalone server or a command tool.
The exportEnapshot can then be moved to the server as well.

Also, thinking about in the higher level.  It is probably beneficial if you
allow HBase to call out an external framework to do computation. It can be
think of as a UDF, a distributed UDF.
The execution of this UDF is totally in separate address spaces, and you
only need to poll the status.  This would be like a dream in traditional
database.

My 2 cents.

Jerry


On Fri, Sep 23, 2016 at 6:46 AM, Matteo Bertozzi 
wrote:

> let me try to go back to my original topic.
> this question was meant to be generic, and provide some rule for future
> code.
>
> from what I can gather, a rule that may satisfy everyone can be:
>  - we don't want any core feature (e.g. compaction/log-split/log-reply)
> over MR, because some cluster may not want or may have an
> external/uncontrolled MR setup.
>  - we allow non-core features (e.g. features enabled by a flag) to run MR
> jobs from hbase, because unless you use the feature, MR is not required.
>
> Matteo
>
>
> On Fri, Sep 23, 2016 at 5:39 AM, Ted Yu  wrote:
>
> > I suggest you look at Matteo's work for AssignmentManager which is to
> make
> > Master more stable.
> >
> > Cheers
> >
> > On Fri, Sep 23, 2016 at 5:32 AM, 张铎  wrote:
> >
> > > No, not your fault, at lease, not this time:)
> > >
> > > Why I call the code ugly? Can you simply tell me the sequence of calls
> > when
> > > starting up the HMaster? HMaster is also a regionserver so it extends
> > > HRegionServer, and the initialization of HRegionServer sometimes needs
> to
> > > make rpc calls to HMaster. A simple change would cause probabilistic
> dead
> > > lock or some strange NPEs...
> > >
> > > That's why I'm very nervous when somebody wants to add new features or
> > add
> > > external dependencies to HMaster, especially add more works for the
> start
> > > up processing...
> > >
> > > Thanks.
> > >
> > > 2016-09-23 20:02 GMT+08:00 Ted Yu :
> > >
> > > > I read through HADOOP-13433
> > > >  - the cited
> race
> > > > condition is in jdk.
> > > >
> > > > Suggest pinging the reviewer on JIRA to get it moving.
> > > >
> > > > bq. But the ugly code in HMaster is readlly a problem...
> > > >
> > > > Can you be specific as to which code is ugly ? Is it in the backup /
> > > > restore mega patch ?
> > > >
> > > > Cheers
> > > >
> > > > On Thu, Sep 22, 2016 at 10:44 PM, 张铎  wrote:
> > > >
> > > > > If you guys have already implemented the feature in the MR way and
> > the
> > > > > patch is ready for landing on master, I'm a -0 on it as I do not
> want
> > > to
> > > > > block the development progress.
> > > > >
> > > > > But I strongly suggest later we need to revisit the design and see
> if
> > > we
> > > > > can seperated the logic from HMaster as much as possible. HA is
> not a
> > > big
> > > > > problem if you do not store any metada locally. But the ugly code
> in
> > > > > HMaster is readlly a problem...
> > > > >
> > > > > And for security, I have a issue pending for a long time. Can
> someone
> > > > help
> > > > > taking a simple look at it? This is what I mean, ugly code...
> logout
> > > and
> > > > > destroy the credentials in a subject when it is still being used,
> and
> > > > > declared as LimitPrivacy so I can not change the behivor and the
> only
> > > way
> > > > > to fix it is to write another piece of ugly code...
> > > > >
> > > > > https://issues.apache.org/jira/browse/HADOOP-13433
> > > > >
> > > > > 2016-09-23 12:53 GMT+08:00 Vladimir Rodionov <
> vladrodio...@gmail.com
> > >:
> > > > >
> > > > > > >> If in the future, we find better ways of doing this without
> > using
> > > > MR,
> > > > > we
> > > > > > can certainly consider that
> > > > > >
> > > > > > Our framework for distributed operations is abstract and allows
> > > > > > different implementations. MR is just one implementation we
> > provide.
> > > > > >
> > > > > > -Vlad
> > > > > >
> > > > > > On Thu, Sep 22, 2016 at 9:38 PM, Devaraj Das <
> d...@hortonworks.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Guys, first off apologies for bringing in the topic of MR-based
> > > > > > > compactions.. But I was thinking more about the SpliceMachine
> > > > approach
> > > > > of
> > > > > > > managing compactions in Spark where apparently they saw a lot
> of
> > > > > > benefits.
> > > > > > > Apologies for giving you that sore throat Andrew; I real

[jira] [Created] (HBASE-16697) bump TestRegionServerMetrics to LargeTests

[Matteo Bertozzi (JIRA)]

Matteo Bertozzi created HBASE-16697:
---

 Summary: bump TestRegionServerMetrics to LargeTests
 Key: HBASE-16697
 URL: https://issues.apache.org/jira/browse/HBASE-16697
 Project: HBase
  Issue Type: Bug
  Components: test
Affects Versions: 2.0.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Trivial
 Fix For: 2.0.0


TestRegionServerMetrics keeps failing because it exceed the MediumTests time 
limit. bump it to Large



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-16292) Can't start a local instance

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He resolved HBASE-16292.
--
Resolution: Duplicate

> Can't start a local instance
> 
>
> Key: HBASE-16292
> URL: https://issues.apache.org/jira/browse/HBASE-16292
> Project: HBase
>  Issue Type: Bug
>Reporter: Elliott Clark
>Priority: Blocker
> Fix For: 2.0.0
>
>
> Currently master is completely broken.
> {code}
> ./bin/start-hbase.sh
> starting master, logging to 
> /Users/elliott/Code/Public/hbase/bin/../logs/hbase-elliott-master-elliott-mbp.out
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; 
> support was removed in 8.0
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; 
> support was removed in 8.0
> {code}
> {code}
> 2016-07-27 10:36:20,992 ERROR [main] regionserver.SecureBulkLoadManager: 
> Failed to create or set permission on staging directory 
> /user/elliott/hbase-staging
> ExitCodeException exitCode=1: chmod: /user/elliott/hbase-staging: No such 
> file or directory
>   at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
>   at org.apache.hadoop.util.Shell.run(Shell.java:456)
>   at 
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
>   at org.apache.hadoop.util.Shell.execCommand(Shell.java:815)
>   at org.apache.hadoop.util.Shell.execCommand(Shell.java:798)
>   at 
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:728)
>   at 
> org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:502)
>   at 
> org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager.start(SecureBulkLoadManager.java:124)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:624)
>   at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:403)
>   at 
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.(HMasterCommandLine.java:307)
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:140)
>   at 
> org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:221)
>   at 
> org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:156)
>   at 
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:226)
>   at 
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>   at 
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:127)
>   at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2417)
> 2016-07-27 10:36:20,994 ERROR [main] master.HMasterCommandLine: Master exiting
> java.lang.RuntimeException: Failed construction of Master: class 
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMasterchmod: 
> /user/elliott/hbase-staging: No such file or directory
>   at 
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:145)
>   at 
> org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:221)
>   at 
> org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:156)
>   at 
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:226)
>   at 
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>   at 
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:127)
>   at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2417)
> Caused by: java.lang.IllegalStateException: Failed to create or set 
> permission on staging directory /user/elliott/hbase-staging
>   at 
> org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager.start(SecureBulkLoadManager.java:141)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:624)
>   at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:403)
>   at 
> org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.(HMasterCommandLine.java:307)
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccess

Re: [DISCUSSION] MR jobs started by Master or RS

[Devaraj Das]

Great points. I am getting a feeling that we have beaten this topic to death 
and have spent enough and more time on it, and it's time to move on. My 
takeaway is that it's fine to do MR for backup/restore - backup/restore is an 
optional feature - unless you use it, you don't need MR.
Fingers crossed,
Devaraj.

P.S. To Nick's point, if it makes sense to do MR/Spark even for core features 
like Compactions, we should be open to it. But that's a topic for another 
[Friday] DISCUSS thread.


From: Jerry He 
Sent: Friday, September 23, 2016 8:40 AM
To: dev
Subject: Re: [DISCUSSION] MR jobs started by Master or RS

That is the point, Matteo.

Put it another way, there is nothing that stops a user from deploying
custom procedure, custom co-processor that calls out MR job.
The optional feature should satisfy some basic requirements. .e.g. No
impact if not deployed or used.  Limited impact if used.
It can be made with isolated dynamic loading of extra configuration (Yarn),
non-blocking non-occupying on the server handlers, or separate handler.
The impact would mostly be on the overall cluster resources. In this sense,
there is no difference, using another standalone server or a command tool.
The exportEnapshot can then be moved to the server as well.

Also, thinking about in the higher level.  It is probably beneficial if you
allow HBase to call out an external framework to do computation. It can be
think of as a UDF, a distributed UDF.
The execution of this UDF is totally in separate address spaces, and you
only need to poll the status.  This would be like a dream in traditional
database.

My 2 cents.

Jerry


On Fri, Sep 23, 2016 at 6:46 AM, Matteo Bertozzi 
wrote:

> let me try to go back to my original topic.
> this question was meant to be generic, and provide some rule for future
> code.
>
> from what I can gather, a rule that may satisfy everyone can be:
>  - we don't want any core feature (e.g. compaction/log-split/log-reply)
> over MR, because some cluster may not want or may have an
> external/uncontrolled MR setup.
>  - we allow non-core features (e.g. features enabled by a flag) to run MR
> jobs from hbase, because unless you use the feature, MR is not required.
>
> Matteo
>
>
> On Fri, Sep 23, 2016 at 5:39 AM, Ted Yu  wrote:
>
> > I suggest you look at Matteo's work for AssignmentManager which is to
> make
> > Master more stable.
> >
> > Cheers
> >
> > On Fri, Sep 23, 2016 at 5:32 AM, 张铎  wrote:
> >
> > > No, not your fault, at lease, not this time:)
> > >
> > > Why I call the code ugly? Can you simply tell me the sequence of calls
> > when
> > > starting up the HMaster? HMaster is also a regionserver so it extends
> > > HRegionServer, and the initialization of HRegionServer sometimes needs
> to
> > > make rpc calls to HMaster. A simple change would cause probabilistic
> dead
> > > lock or some strange NPEs...
> > >
> > > That's why I'm very nervous when somebody wants to add new features or
> > add
> > > external dependencies to HMaster, especially add more works for the
> start
> > > up processing...
> > >
> > > Thanks.
> > >
> > > 2016-09-23 20:02 GMT+08:00 Ted Yu :
> > >
> > > > I read through HADOOP-13433
> > > >  - the cited
> race
> > > > condition is in jdk.
> > > >
> > > > Suggest pinging the reviewer on JIRA to get it moving.
> > > >
> > > > bq. But the ugly code in HMaster is readlly a problem...
> > > >
> > > > Can you be specific as to which code is ugly ? Is it in the backup /
> > > > restore mega patch ?
> > > >
> > > > Cheers
> > > >
> > > > On Thu, Sep 22, 2016 at 10:44 PM, 张铎  wrote:
> > > >
> > > > > If you guys have already implemented the feature in the MR way and
> > the
> > > > > patch is ready for landing on master, I'm a -0 on it as I do not
> want
> > > to
> > > > > block the development progress.
> > > > >
> > > > > But I strongly suggest later we need to revisit the design and see
> if
> > > we
> > > > > can seperated the logic from HMaster as much as possible. HA is
> not a
> > > big
> > > > > problem if you do not store any metada locally. But the ugly code
> in
> > > > > HMaster is readlly a problem...
> > > > >
> > > > > And for security, I have a issue pending for a long time. Can
> someone
> > > > help
> > > > > taking a simple look at it? This is what I mean, ugly code...
> logout
> > > and
> > > > > destroy the credentials in a subject when it is still being used,
> and
> > > > > declared as LimitPrivacy so I can not change the behivor and the
> only
> > > way
> > > > > to fix it is to write another piece of ugly code...
> > > > >
> > > > > https://issues.apache.org/jira/browse/HADOOP-13433
> > > > >
> > > > > 2016-09-23 12:53 GMT+08:00 Vladimir Rodionov <
> vladrodio...@gmail.com
> > >:
> > > > >
> > > > > > >> If in the future, we find better ways of doing this without
> > using
> > > > MR,
> > > > > we
> > > > > > can certainly consider that
> > > > > >
> > > > > > Our fr

Re: [DISCUSSION] MR jobs started by Master or RS

[Stack]

On Fri, Sep 23, 2016 at 6:46 AM, Matteo Bertozzi 
wrote:

> let me try to go back to my original topic.
> this question was meant to be generic, and provide some rule for future
> code.
>
> from what I can gather, a rule that may satisfy everyone can be:
>  - we don't want any core feature (e.g. compaction/log-split/log-reply)
> over MR, because some cluster may not want or may have an
> external/uncontrolled MR setup.
>

+1


>  - we allow non-core features (e.g. features enabled by a flag) to run MR
> jobs from hbase, because unless you use the feature, MR is not required.
>
>
-1 to hbase core depending on MR or core -- whether behind a flag or not --
ever being able to launch MR jobs.

+ MR is dead. We should be busy working hard to undo it from hbase-server
moving it out to be an optional module (Spark would be its peer).
+ Master is a rats nest of state. Matteo, Stephen, and Appy are busy
working hard on moving it up on to a new foundation. Lets not clutter task
harder by piling on more moving parts.

St.Ack


> Matteo
>
>
> On Fri, Sep 23, 2016 at 5:39 AM, Ted Yu  wrote:
>
> > I suggest you look at Matteo's work for AssignmentManager which is to
> make
> > Master more stable.
> >
> > Cheers
> >
> > On Fri, Sep 23, 2016 at 5:32 AM, 张铎  wrote:
> >
> > > No, not your fault, at lease, not this time:)
> > >
> > > Why I call the code ugly? Can you simply tell me the sequence of calls
> > when
> > > starting up the HMaster? HMaster is also a regionserver so it extends
> > > HRegionServer, and the initialization of HRegionServer sometimes needs
> to
> > > make rpc calls to HMaster. A simple change would cause probabilistic
> dead
> > > lock or some strange NPEs...
> > >
> > > That's why I'm very nervous when somebody wants to add new features or
> > add
> > > external dependencies to HMaster, especially add more works for the
> start
> > > up processing...
> > >
> > > Thanks.
> > >
> > > 2016-09-23 20:02 GMT+08:00 Ted Yu :
> > >
> > > > I read through HADOOP-13433
> > > >  - the cited
> race
> > > > condition is in jdk.
> > > >
> > > > Suggest pinging the reviewer on JIRA to get it moving.
> > > >
> > > > bq. But the ugly code in HMaster is readlly a problem...
> > > >
> > > > Can you be specific as to which code is ugly ? Is it in the backup /
> > > > restore mega patch ?
> > > >
> > > > Cheers
> > > >
> > > > On Thu, Sep 22, 2016 at 10:44 PM, 张铎  wrote:
> > > >
> > > > > If you guys have already implemented the feature in the MR way and
> > the
> > > > > patch is ready for landing on master, I'm a -0 on it as I do not
> want
> > > to
> > > > > block the development progress.
> > > > >
> > > > > But I strongly suggest later we need to revisit the design and see
> if
> > > we
> > > > > can seperated the logic from HMaster as much as possible. HA is
> not a
> > > big
> > > > > problem if you do not store any metada locally. But the ugly code
> in
> > > > > HMaster is readlly a problem...
> > > > >
> > > > > And for security, I have a issue pending for a long time. Can
> someone
> > > > help
> > > > > taking a simple look at it? This is what I mean, ugly code...
> logout
> > > and
> > > > > destroy the credentials in a subject when it is still being used,
> and
> > > > > declared as LimitPrivacy so I can not change the behivor and the
> only
> > > way
> > > > > to fix it is to write another piece of ugly code...
> > > > >
> > > > > https://issues.apache.org/jira/browse/HADOOP-13433
> > > > >
> > > > > 2016-09-23 12:53 GMT+08:00 Vladimir Rodionov <
> vladrodio...@gmail.com
> > >:
> > > > >
> > > > > > >> If in the future, we find better ways of doing this without
> > using
> > > > MR,
> > > > > we
> > > > > > can certainly consider that
> > > > > >
> > > > > > Our framework for distributed operations is abstract and allows
> > > > > > different implementations. MR is just one implementation we
> > provide.
> > > > > >
> > > > > > -Vlad
> > > > > >
> > > > > > On Thu, Sep 22, 2016 at 9:38 PM, Devaraj Das <
> d...@hortonworks.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Guys, first off apologies for bringing in the topic of MR-based
> > > > > > > compactions.. But I was thinking more about the SpliceMachine
> > > > approach
> > > > > of
> > > > > > > managing compactions in Spark where apparently they saw a lot
> of
> > > > > > benefits.
> > > > > > > Apologies for giving you that sore throat Andrew; I really
> didn't
> > > > mean
> > > > > to
> > > > > > > :-)
> > > > > > >
> > > > > > > So on this issue, we have these on the plate:
> > > > > > > 0. Somehow not use MR but something like that
> > > > > > > 1. Run a standalone service other than master
> > > > > > > 2. Shell out from the master
> > > > > > >
> > > > > > > I don't think we have a good answer to (0), and I don't think
> > it's
> > > > even
> > > > > > > worth the effort of trying to build something when MR is
> already
> > > > there,
> > > > > > and
> > > > > > > being used by HBase already

Re: [DISCUSSION] MR jobs started by Master or RS

[Vladimir Rodionov]

>> + MR is dead

Does MR know that? :)

Again. With all due respect, stack - still no suggestions what should we
use for "bulk data move and transformation" instead of MR?
Backup needs this, by default. We are talking about TBs of data in a single
session. We not only copy data between src and dest,
but transform it as well on-the-fly (WALs into HFiles)

That can be Spark in a future, of course, but not right now. Framework is
open.

I suggest voting first on "do we need backup in HBase"? In my opinion, some
group members still not sure about that and some will give -1
in any case. Just because ...

-Vlad






On Fri, Sep 23, 2016 at 10:57 AM, Stack  wrote:

> On Fri, Sep 23, 2016 at 6:46 AM, Matteo Bertozzi 
> wrote:
>
> > let me try to go back to my original topic.
> > this question was meant to be generic, and provide some rule for future
> > code.
> >
> > from what I can gather, a rule that may satisfy everyone can be:
> >  - we don't want any core feature (e.g. compaction/log-split/log-reply)
> > over MR, because some cluster may not want or may have an
> > external/uncontrolled MR setup.
> >
>
> +1
>
>
> >  - we allow non-core features (e.g. features enabled by a flag) to run MR
> > jobs from hbase, because unless you use the feature, MR is not required.
> >
> >
> -1 to hbase core depending on MR or core -- whether behind a flag or not --
> ever being able to launch MR jobs.
>
> + MR is dead. We should be busy working hard to undo it from hbase-server
> moving it out to be an optional module (Spark would be its peer).
> + Master is a rats nest of state. Matteo, Stephen, and Appy are busy
> working hard on moving it up on to a new foundation. Lets not clutter task
> harder by piling on more moving parts.
>
> St.Ack
>
>
> > Matteo
> >
> >
> > On Fri, Sep 23, 2016 at 5:39 AM, Ted Yu  wrote:
> >
> > > I suggest you look at Matteo's work for AssignmentManager which is to
> > make
> > > Master more stable.
> > >
> > > Cheers
> > >
> > > On Fri, Sep 23, 2016 at 5:32 AM, 张铎  wrote:
> > >
> > > > No, not your fault, at lease, not this time:)
> > > >
> > > > Why I call the code ugly? Can you simply tell me the sequence of
> calls
> > > when
> > > > starting up the HMaster? HMaster is also a regionserver so it extends
> > > > HRegionServer, and the initialization of HRegionServer sometimes
> needs
> > to
> > > > make rpc calls to HMaster. A simple change would cause probabilistic
> > dead
> > > > lock or some strange NPEs...
> > > >
> > > > That's why I'm very nervous when somebody wants to add new features
> or
> > > add
> > > > external dependencies to HMaster, especially add more works for the
> > start
> > > > up processing...
> > > >
> > > > Thanks.
> > > >
> > > > 2016-09-23 20:02 GMT+08:00 Ted Yu :
> > > >
> > > > > I read through HADOOP-13433
> > > > >  - the cited
> > race
> > > > > condition is in jdk.
> > > > >
> > > > > Suggest pinging the reviewer on JIRA to get it moving.
> > > > >
> > > > > bq. But the ugly code in HMaster is readlly a problem...
> > > > >
> > > > > Can you be specific as to which code is ugly ? Is it in the backup
> /
> > > > > restore mega patch ?
> > > > >
> > > > > Cheers
> > > > >
> > > > > On Thu, Sep 22, 2016 at 10:44 PM, 张铎 
> wrote:
> > > > >
> > > > > > If you guys have already implemented the feature in the MR way
> and
> > > the
> > > > > > patch is ready for landing on master, I'm a -0 on it as I do not
> > want
> > > > to
> > > > > > block the development progress.
> > > > > >
> > > > > > But I strongly suggest later we need to revisit the design and
> see
> > if
> > > > we
> > > > > > can seperated the logic from HMaster as much as possible. HA is
> > not a
> > > > big
> > > > > > problem if you do not store any metada locally. But the ugly code
> > in
> > > > > > HMaster is readlly a problem...
> > > > > >
> > > > > > And for security, I have a issue pending for a long time. Can
> > someone
> > > > > help
> > > > > > taking a simple look at it? This is what I mean, ugly code...
> > logout
> > > > and
> > > > > > destroy the credentials in a subject when it is still being used,
> > and
> > > > > > declared as LimitPrivacy so I can not change the behivor and the
> > only
> > > > way
> > > > > > to fix it is to write another piece of ugly code...
> > > > > >
> > > > > > https://issues.apache.org/jira/browse/HADOOP-13433
> > > > > >
> > > > > > 2016-09-23 12:53 GMT+08:00 Vladimir Rodionov <
> > vladrodio...@gmail.com
> > > >:
> > > > > >
> > > > > > > >> If in the future, we find better ways of doing this without
> > > using
> > > > > MR,
> > > > > > we
> > > > > > > can certainly consider that
> > > > > > >
> > > > > > > Our framework for distributed operations is abstract and allows
> > > > > > > different implementations. MR is just one implementation we
> > > provide.
> > > > > > >
> > > > > > > -Vlad
> > > > > > >
> > > > > > > On Thu, Sep 22, 2016 at 9:38 PM, Devaraj Das <
> > d...@hortonworks.com
> > > >
> > 

Re: [DISCUSSION] MR jobs started by Master or RS

[Stack]

On Fri, Sep 23, 2016 at 11:22 AM, Vladimir Rodionov 
wrote:

> >> + MR is dead
>
> Does MR know that? :)
>
> Again. With all due respect, stack - still no suggestions what should we
> use for "bulk data move and transformation" instead of MR?
>

Use whatever distributed engine suits your fancy -- MR, Spark, distributed
shell -- just don't have HBase core depend on it, even optionally.


> I suggest voting first on "do we need backup in HBase"? In my opinion, some
> group members still not sure about that and some will give -1
> in any case. Just because ...
>
>
We could run a vote, sure. -1 on that backup be in core hbase (+1 on adding
all the API any such external tool might need to run).

St.Ack



> -Vlad
>
>
>
>
>
>
> On Fri, Sep 23, 2016 at 10:57 AM, Stack  wrote:
>
> > On Fri, Sep 23, 2016 at 6:46 AM, Matteo Bertozzi <
> theo.berto...@gmail.com>
> > wrote:
> >
> > > let me try to go back to my original topic.
> > > this question was meant to be generic, and provide some rule for future
> > > code.
> > >
> > > from what I can gather, a rule that may satisfy everyone can be:
> > >  - we don't want any core feature (e.g. compaction/log-split/log-
> reply)
> > > over MR, because some cluster may not want or may have an
> > > external/uncontrolled MR setup.
> > >
> >
> > +1
> >
> >
> > >  - we allow non-core features (e.g. features enabled by a flag) to run
> MR
> > > jobs from hbase, because unless you use the feature, MR is not
> required.
> > >
> > >
> > -1 to hbase core depending on MR or core -- whether behind a flag or not
> --
> > ever being able to launch MR jobs.
> >
> > + MR is dead. We should be busy working hard to undo it from hbase-server
> > moving it out to be an optional module (Spark would be its peer).
> > + Master is a rats nest of state. Matteo, Stephen, and Appy are busy
> > working hard on moving it up on to a new foundation. Lets not clutter
> task
> > harder by piling on more moving parts.
> >
> > St.Ack
> >
> >
> > > Matteo
> > >
> > >
> > > On Fri, Sep 23, 2016 at 5:39 AM, Ted Yu  wrote:
> > >
> > > > I suggest you look at Matteo's work for AssignmentManager which is to
> > > make
> > > > Master more stable.
> > > >
> > > > Cheers
> > > >
> > > > On Fri, Sep 23, 2016 at 5:32 AM, 张铎  wrote:
> > > >
> > > > > No, not your fault, at lease, not this time:)
> > > > >
> > > > > Why I call the code ugly? Can you simply tell me the sequence of
> > calls
> > > > when
> > > > > starting up the HMaster? HMaster is also a regionserver so it
> extends
> > > > > HRegionServer, and the initialization of HRegionServer sometimes
> > needs
> > > to
> > > > > make rpc calls to HMaster. A simple change would cause
> probabilistic
> > > dead
> > > > > lock or some strange NPEs...
> > > > >
> > > > > That's why I'm very nervous when somebody wants to add new features
> > or
> > > > add
> > > > > external dependencies to HMaster, especially add more works for the
> > > start
> > > > > up processing...
> > > > >
> > > > > Thanks.
> > > > >
> > > > > 2016-09-23 20:02 GMT+08:00 Ted Yu :
> > > > >
> > > > > > I read through HADOOP-13433
> > > > > >  - the cited
> > > race
> > > > > > condition is in jdk.
> > > > > >
> > > > > > Suggest pinging the reviewer on JIRA to get it moving.
> > > > > >
> > > > > > bq. But the ugly code in HMaster is readlly a problem...
> > > > > >
> > > > > > Can you be specific as to which code is ugly ? Is it in the
> backup
> > /
> > > > > > restore mega patch ?
> > > > > >
> > > > > > Cheers
> > > > > >
> > > > > > On Thu, Sep 22, 2016 at 10:44 PM, 张铎 
> > wrote:
> > > > > >
> > > > > > > If you guys have already implemented the feature in the MR way
> > and
> > > > the
> > > > > > > patch is ready for landing on master, I'm a -0 on it as I do
> not
> > > want
> > > > > to
> > > > > > > block the development progress.
> > > > > > >
> > > > > > > But I strongly suggest later we need to revisit the design and
> > see
> > > if
> > > > > we
> > > > > > > can seperated the logic from HMaster as much as possible. HA is
> > > not a
> > > > > big
> > > > > > > problem if you do not store any metada locally. But the ugly
> code
> > > in
> > > > > > > HMaster is readlly a problem...
> > > > > > >
> > > > > > > And for security, I have a issue pending for a long time. Can
> > > someone
> > > > > > help
> > > > > > > taking a simple look at it? This is what I mean, ugly code...
> > > logout
> > > > > and
> > > > > > > destroy the credentials in a subject when it is still being
> used,
> > > and
> > > > > > > declared as LimitPrivacy so I can not change the behivor and
> the
> > > only
> > > > > way
> > > > > > > to fix it is to write another piece of ugly code...
> > > > > > >
> > > > > > > https://issues.apache.org/jira/browse/HADOOP-13433
> > > > > > >
> > > > > > > 2016-09-23 12:53 GMT+08:00 Vladimir Rodionov <
> > > vladrodio...@gmail.com
> > > > >:
> > > > > > >
> > > > > > > > >> If in the future, we find better ways of doing this

[jira] [Resolved] (HBASE-16693) The commit of HBASE-16604 creates a unrelated folder

[Enis Soztutar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Enis Soztutar resolved HBASE-16693.
---
   Resolution: Fixed
Fix Version/s: 2.0.0

Should be resolved now. 

> The commit of HBASE-16604 creates a unrelated folder
> 
>
> Key: HBASE-16693
> URL: https://issues.apache.org/jira/browse/HBASE-16693
> Project: HBase
>  Issue Type: Bug
>Reporter: ChiaPing Tsai
>Priority: Critical
> Fix For: 2.0.0
>
>
> The folder's name is "hbase-thrift\036src", and it causes the git-clone 
> failure on windows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSSION] MR jobs started by Master or RS

[Vladimir Rodionov]

>>  -1 on that backup be in core hbase

Not sure I understand what it means.

1. We are not allowed to use Master to orchestrate the whole process? We
have already brought up all advantages of using
   Master and distributed procedures for backup and restore.


Downside of moving this to client tool is lack of fault tolerance:
 1.1 Client won't be allowed to do any operations, that can, potentially affect
cluster, such as disabling splits/merges, balancer.
 1.2 In case of client failure who will be doing the whole rollback stuff?
We are trying to make it atomic.

Security is not clear.

2. We are not allowed to modify code of existing HBase core classes (what
does core mean anyway)?

3. We are not allowed to create backup system table (hbase:backup) in a
system space? Only in user space? The table is global.

2. is critical. Despite the fact, that 95% of code is new, we have touched,
of course some existing HBase code.
3. is not that critical, of course we can move backup system into user
space.

And finally, will moving backup into external tool give us +1 from stack?

-Vlad





On Fri, Sep 23, 2016 at 11:26 AM, Stack  wrote:

> On Fri, Sep 23, 2016 at 11:22 AM, Vladimir Rodionov <
> vladrodio...@gmail.com>
> wrote:
>
> > >> + MR is dead
> >
> > Does MR know that? :)
> >
> > Again. With all due respect, stack - still no suggestions what should we
> > use for "bulk data move and transformation" instead of MR?
> >
>
> Use whatever distributed engine suits your fancy -- MR, Spark, distributed
> shell -- just don't have HBase core depend on it, even optionally.
>
>
> > I suggest voting first on "do we need backup in HBase"? In my opinion,
> some
> > group members still not sure about that and some will give -1
> > in any case. Just because ...
> >
> >
> We could run a vote, sure. -1 on that backup be in core hbase (+1 on adding
> all the API any such external tool might need to run).
>
> St.Ack
>
>
>
> > -Vlad
> >
> >
> >
> >
> >
> >
> > On Fri, Sep 23, 2016 at 10:57 AM, Stack  wrote:
> >
> > > On Fri, Sep 23, 2016 at 6:46 AM, Matteo Bertozzi <
> > theo.berto...@gmail.com>
> > > wrote:
> > >
> > > > let me try to go back to my original topic.
> > > > this question was meant to be generic, and provide some rule for
> future
> > > > code.
> > > >
> > > > from what I can gather, a rule that may satisfy everyone can be:
> > > >  - we don't want any core feature (e.g. compaction/log-split/log-
> > reply)
> > > > over MR, because some cluster may not want or may have an
> > > > external/uncontrolled MR setup.
> > > >
> > >
> > > +1
> > >
> > >
> > > >  - we allow non-core features (e.g. features enabled by a flag) to
> run
> > MR
> > > > jobs from hbase, because unless you use the feature, MR is not
> > required.
> > > >
> > > >
> > > -1 to hbase core depending on MR or core -- whether behind a flag or
> not
> > --
> > > ever being able to launch MR jobs.
> > >
> > > + MR is dead. We should be busy working hard to undo it from
> hbase-server
> > > moving it out to be an optional module (Spark would be its peer).
> > > + Master is a rats nest of state. Matteo, Stephen, and Appy are busy
> > > working hard on moving it up on to a new foundation. Lets not clutter
> > task
> > > harder by piling on more moving parts.
> > >
> > > St.Ack
> > >
> > >
> > > > Matteo
> > > >
> > > >
> > > > On Fri, Sep 23, 2016 at 5:39 AM, Ted Yu  wrote:
> > > >
> > > > > I suggest you look at Matteo's work for AssignmentManager which is
> to
> > > > make
> > > > > Master more stable.
> > > > >
> > > > > Cheers
> > > > >
> > > > > On Fri, Sep 23, 2016 at 5:32 AM, 张铎  wrote:
> > > > >
> > > > > > No, not your fault, at lease, not this time:)
> > > > > >
> > > > > > Why I call the code ugly? Can you simply tell me the sequence of
> > > calls
> > > > > when
> > > > > > starting up the HMaster? HMaster is also a regionserver so it
> > extends
> > > > > > HRegionServer, and the initialization of HRegionServer sometimes
> > > needs
> > > > to
> > > > > > make rpc calls to HMaster. A simple change would cause
> > probabilistic
> > > > dead
> > > > > > lock or some strange NPEs...
> > > > > >
> > > > > > That's why I'm very nervous when somebody wants to add new
> features
> > > or
> > > > > add
> > > > > > external dependencies to HMaster, especially add more works for
> the
> > > > start
> > > > > > up processing...
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > 2016-09-23 20:02 GMT+08:00 Ted Yu :
> > > > > >
> > > > > > > I read through HADOOP-13433
> > > > > > >  - the
> cited
> > > > race
> > > > > > > condition is in jdk.
> > > > > > >
> > > > > > > Suggest pinging the reviewer on JIRA to get it moving.
> > > > > > >
> > > > > > > bq. But the ugly code in HMaster is readlly a problem...
> > > > > > >
> > > > > > > Can you be specific as to which code is ugly ? Is it in the
> > backup
> > > /
> > > > > > > restore mega patch ?
> > > > > > >
> > > > > > > Cheers
> > > > > > >
> > > >

[jira] [Created] (HBASE-16698) Handlers stuck waiting for CountDownLatch inside WALKey#getWriteEntry under high writing workload

[Yu Li (JIRA)]

Yu Li created HBASE-16698:
-

 Summary: Handlers stuck waiting for CountDownLatch inside 
WALKey#getWriteEntry under high writing workload
 Key: HBASE-16698
 URL: https://issues.apache.org/jira/browse/HBASE-16698
 Project: HBase
  Issue Type: Improvement
  Components: Performance
Affects Versions: 1.2.3, 1.1.6
Reporter: Yu Li
Assignee: Yu Li


As titled, on our production environment we observed 98 out of 128 handlers get 
stuck waiting for the CountDownLatch {{seqNumAssignedLatch}} inside 
{{WALKey#getWriteEntry}} under a high writing workload.

After digging into the problem, we found that the problem is mainly caused by 
advancing mvcc in the append logic. Below is some detailed analysis:

Under current branch-1 code logic, all batch puts will call 
{{WALKey#getWriteEntry}} after appending edit to WAL, and 
{{seqNumAssignedLatch}} is only released when the relative append call is 
handled by RingBufferEventHandler (see {{FSWALEntry#stampRegionSequenceId}}). 
Because currently we're using a single event handler for the ringbuffer, the 
append calls are handled one by one (actually lot's of our current logic 
depending on this sequential dealing logic), and this becomes a bottleneck 
under high writing workload.

The worst part is that by default we only use one WAL per RS, so appends on all 
regions are dealt with in sequential, which causes contention among different 
regions...

To fix this, we could also take use of the "sequential appends" mechanism, that 
we could grab the WriteEntry before publishing append onto ringbuffer and use 
it as sequence id, only that we need to add a lock to make "grab WriteEntry" 
and "append edit" a transaction. This will still cause contention inside a 
region but could avoid contention between different regions. This solution is 
already verified in our online environment and proved to be effective.

Notice that for master (2.0) branch since we already change the write pipeline 
to sync before writing memstore (HBASE-15158), this issue only exists for the 
ASYNC_WAL writes scenario.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16699) Overflows in AverageIntervalRateLimiter's refill() and getWaitInterval()

[huaxiang sun (JIRA)]

huaxiang sun created HBASE-16699:


 Summary: Overflows in AverageIntervalRateLimiter's refill() and 
getWaitInterval()
 Key: HBASE-16699
 URL: https://issues.apache.org/jira/browse/HBASE-16699
 Project: HBase
  Issue Type: Bug
Affects Versions: 2.0.0, 1.4.0
Reporter: huaxiang sun
Assignee: huaxiang sun


Please see the following two lines. Once it overflows, it will cause wrong 
behavior. For unconfigured RateLimiters, we should have simpler logic to byPass 
the check. 

https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/AverageIntervalRateLimiter.java#L37

https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/AverageIntervalRateLimiter.java#L51



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16700) Allow for coprocessor whitelisting

[Clay B. (JIRA)]

Clay B. created HBASE-16700:
---

 Summary: Allow for coprocessor whitelisting
 Key: HBASE-16700
 URL: https://issues.apache.org/jira/browse/HBASE-16700
 Project: HBase
  Issue Type: Improvement
  Components: Coprocessors
Reporter: Clay B.
Priority: Minor


Today one can turn off all non-system coprocessors with 
{{hbase.coprocessor.user.enabled}} however, this disables very useful things 
like Apache Phoenix's coprocessors. Some tenants of a multi-user HBase may also 
need to run bespoke coprocessors. But as an operator I would not want wanton 
coprocessor usage. Ideally, one could do one of two things:
* Allow coprocessors defined in {{hbase-site.xml}} -- this can only be 
administratively changed in most cases
* Allow coprocessors from table descriptors but only if the coprocessor is 
whitelisted



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSSION] MR jobs started by Master or RS

[Andrew Purtell]

It's simple: Use a tool (something derived from Tool, like our other
utilities for import, export, etc.), or use a separate process, but don't
"shell out" to MR from the Master.


On Fri, Sep 23, 2016 at 12:24 PM, Vladimir Rodionov 
wrote:

> >>  -1 on that backup be in core hbase
>
> Not sure I understand what it means.
>
> 1. We are not allowed to use Master to orchestrate the whole process? We
> have already brought up all advantages of using
>Master and distributed procedures for backup and restore.
>
>
> Downside of moving this to client tool is lack of fault tolerance:
>  1.1 Client won't be allowed to do any operations, that can, potentially
> affect
> cluster, such as disabling splits/merges, balancer.
>  1.2 In case of client failure who will be doing the whole rollback stuff?
> We are trying to make it atomic.
>
> Security is not clear.
>
> 2. We are not allowed to modify code of existing HBase core classes (what
> does core mean anyway)?
>
> 3. We are not allowed to create backup system table (hbase:backup) in a
> system space? Only in user space? The table is global.
>
> 2. is critical. Despite the fact, that 95% of code is new, we have touched,
> of course some existing HBase code.
> 3. is not that critical, of course we can move backup system into user
> space.
>
> And finally, will moving backup into external tool give us +1 from stack?
>
> -Vlad
>
>
>
>
>
> On Fri, Sep 23, 2016 at 11:26 AM, Stack  wrote:
>
> > On Fri, Sep 23, 2016 at 11:22 AM, Vladimir Rodionov <
> > vladrodio...@gmail.com>
> > wrote:
> >
> > > >> + MR is dead
> > >
> > > Does MR know that? :)
> > >
> > > Again. With all due respect, stack - still no suggestions what should
> we
> > > use for "bulk data move and transformation" instead of MR?
> > >
> >
> > Use whatever distributed engine suits your fancy -- MR, Spark,
> distributed
> > shell -- just don't have HBase core depend on it, even optionally.
> >
> >
> > > I suggest voting first on "do we need backup in HBase"? In my opinion,
> > some
> > > group members still not sure about that and some will give -1
> > > in any case. Just because ...
> > >
> > >
> > We could run a vote, sure. -1 on that backup be in core hbase (+1 on
> adding
> > all the API any such external tool might need to run).
> >
> > St.Ack
> >
> >
> >
> > > -Vlad
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Fri, Sep 23, 2016 at 10:57 AM, Stack  wrote:
> > >
> > > > On Fri, Sep 23, 2016 at 6:46 AM, Matteo Bertozzi <
> > > theo.berto...@gmail.com>
> > > > wrote:
> > > >
> > > > > let me try to go back to my original topic.
> > > > > this question was meant to be generic, and provide some rule for
> > future
> > > > > code.
> > > > >
> > > > > from what I can gather, a rule that may satisfy everyone can be:
> > > > >  - we don't want any core feature (e.g. compaction/log-split/log-
> > > reply)
> > > > > over MR, because some cluster may not want or may have an
> > > > > external/uncontrolled MR setup.
> > > > >
> > > >
> > > > +1
> > > >
> > > >
> > > > >  - we allow non-core features (e.g. features enabled by a flag) to
> > run
> > > MR
> > > > > jobs from hbase, because unless you use the feature, MR is not
> > > required.
> > > > >
> > > > >
> > > > -1 to hbase core depending on MR or core -- whether behind a flag or
> > not
> > > --
> > > > ever being able to launch MR jobs.
> > > >
> > > > + MR is dead. We should be busy working hard to undo it from
> > hbase-server
> > > > moving it out to be an optional module (Spark would be its peer).
> > > > + Master is a rats nest of state. Matteo, Stephen, and Appy are busy
> > > > working hard on moving it up on to a new foundation. Lets not clutter
> > > task
> > > > harder by piling on more moving parts.
> > > >
> > > > St.Ack
> > > >
> > > >
> > > > > Matteo
> > > > >
> > > > >
> > > > > On Fri, Sep 23, 2016 at 5:39 AM, Ted Yu 
> wrote:
> > > > >
> > > > > > I suggest you look at Matteo's work for AssignmentManager which
> is
> > to
> > > > > make
> > > > > > Master more stable.
> > > > > >
> > > > > > Cheers
> > > > > >
> > > > > > On Fri, Sep 23, 2016 at 5:32 AM, 张铎 
> wrote:
> > > > > >
> > > > > > > No, not your fault, at lease, not this time:)
> > > > > > >
> > > > > > > Why I call the code ugly? Can you simply tell me the sequence
> of
> > > > calls
> > > > > > when
> > > > > > > starting up the HMaster? HMaster is also a regionserver so it
> > > extends
> > > > > > > HRegionServer, and the initialization of HRegionServer
> sometimes
> > > > needs
> > > > > to
> > > > > > > make rpc calls to HMaster. A simple change would cause
> > > probabilistic
> > > > > dead
> > > > > > > lock or some strange NPEs...
> > > > > > >
> > > > > > > That's why I'm very nervous when somebody wants to add new
> > features
> > > > or
> > > > > > add
> > > > > > > external dependencies to HMaster, especially add more works for
> > the
> > > > > start
> > > > > > > up processing...
> > > > > > >
> > > > > > > Thanks.
> > > > > > >
> > > > > > > 2016-09-23 20:02 GMT+08:

[jira] [Created] (HBASE-16701) TestHRegion and TestHRegionWithInMemoryFlush timing out

[Appy (JIRA)]

Appy created HBASE-16701:


 Summary: TestHRegion and TestHRegionWithInMemoryFlush timing out
 Key: HBASE-16701
 URL: https://issues.apache.org/jira/browse/HBASE-16701
 Project: HBase
  Issue Type: Bug
Reporter: Appy
Assignee: Appy


These two tests are marked LargeTests and thus have a timeout of 10 minutes, 
but it's not sufficient since they have like more than 100 testcases.
Ideal fix would be splitting the tests, but that'll be much more involved 
effort.
For now, increasing the timeout to 15min.

Current stats:
On apache infra:
TestHRegionWithInMemoryFlush20.0% (15 / 75) 15 / 12 / 0
TestHRegion 9.3% (7 / 75)   7 / 5 / 0

On GCE infra:
TestHRegion 89.7% (87 / 97) 87 / 87 / 0
TestHRegionWithInMemoryFlush71.1% (69 / 97) 69 / 69 / 0




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Backup Implementation (WAS => Re: [DISCUSSION] MR jobs started by Master or RS)

[Stack]

(Moved out of the Master doing MR DISCUSSION)

On Fri, Sep 23, 2016 at 12:24 PM, Vladimir Rodionov 
wrote:

> >>  -1 on that backup be in core hbase
>
> Not sure I understand what it means.
>
> Sorry for the imprecision.

The -1 is NOT against backup/restore. I am -1 on MR as a dependency and so
-1 on the Master running backup/restore MR jobs, even if optional.

Master should not depend on MR. We've gone out of our way to avoid taking
MR on as dependency in the past. Seems late in the game for us to change
our opinion on this. If we didn't do it for distributed log splitting, or
MOB, why would we do it to support an optional backup/restore?

I have opinions on the questions below -- i.e. that Master running
backup/restore is outside of the Master's charge -- but they are not worth
much since I've not done much by way of review or contrib to backup/restore
other than to try it as a 'user' so I'll keep them to myself until I do. I
only came out from under my shell to participate on the MR as dependency
chat.

Thanks,
M


1. We are not allowed to use Master to orchestrate the whole process?


We
> have already brought up all advantages of using
>Master and distributed procedures for backup and restore.
>
>
> Downside of moving this to client tool is lack of fault tolerance:
>  1.1 Client won't be allowed to do any operations, that can, potentially
> affect
> cluster, such as disabling splits/merges, balancer.
>  1.2 In case of client failure who will be doing the whole rollback stuff?
> We are trying to make it atomic.
>
> Security is not clear.



2. We are not allowed to modify code of existing HBase core classes (what
> does core mean anyway)?
>
>


> 3. We are not allowed to create backup system table (hbase:backup) in a
> system space? Only in user space? The table is global.
>


> 2. is critical. Despite the fact, that 95% of code is new, we have touched,
> of course some existing HBase code.
> 3. is not that critical, of course we can move backup system into user
> space.
>
> And finally, will moving backup into external tool give us +1 from stack?
>
> -Vlad
>
>
>
>
>
> On Fri, Sep 23, 2016 at 11:26 AM, Stack  wrote:
>
> > On Fri, Sep 23, 2016 at 11:22 AM, Vladimir Rodionov <
> > vladrodio...@gmail.com>
> > wrote:
> >
> > > >> + MR is dead
> > >
> > > Does MR know that? :)
> > >
> > > Again. With all due respect, stack - still no suggestions what should
> we
> > > use for "bulk data move and transformation" instead of MR?
> > >
> >
> > Use whatever distributed engine suits your fancy -- MR, Spark,
> distributed
> > shell -- just don't have HBase core depend on it, even optionally.
> >
> >
> > > I suggest voting first on "do we need backup in HBase"? In my opinion,
> > some
> > > group members still not sure about that and some will give -1
> > > in any case. Just because ...
> > >
> > >
> > We could run a vote, sure. -1 on that backup be in core hbase (+1 on
> adding
> > all the API any such external tool might need to run).
> >
> > St.Ack
> >
> >
> >
> > > -Vlad
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Fri, Sep 23, 2016 at 10:57 AM, Stack  wrote:
> > >
> > > > On Fri, Sep 23, 2016 at 6:46 AM, Matteo Bertozzi <
> > > theo.berto...@gmail.com>
> > > > wrote:
> > > >
> > > > > let me try to go back to my original topic.
> > > > > this question was meant to be generic, and provide some rule for
> > future
> > > > > code.
> > > > >
> > > > > from what I can gather, a rule that may satisfy everyone can be:
> > > > >  - we don't want any core feature (e.g. compaction/log-split/log-
> > > reply)
> > > > > over MR, because some cluster may not want or may have an
> > > > > external/uncontrolled MR setup.
> > > > >
> > > >
> > > > +1
> > > >
> > > >
> > > > >  - we allow non-core features (e.g. features enabled by a flag) to
> > run
> > > MR
> > > > > jobs from hbase, because unless you use the feature, MR is not
> > > required.
> > > > >
> > > > >
> > > > -1 to hbase core depending on MR or core -- whether behind a flag or
> > not
> > > --
> > > > ever being able to launch MR jobs.
> > > >
> > > > + MR is dead. We should be busy working hard to undo it from
> > hbase-server
> > > > moving it out to be an optional module (Spark would be its peer).
> > > > + Master is a rats nest of state. Matteo, Stephen, and Appy are busy
> > > > working hard on moving it up on to a new foundation. Lets not clutter
> > > task
> > > > harder by piling on more moving parts.
> > > >
> > > > St.Ack
> > > >
> > > >
> > > > > Matteo
> > > > >
> > > > >
> > > > > On Fri, Sep 23, 2016 at 5:39 AM, Ted Yu 
> wrote:
> > > > >
> > > > > > I suggest you look at Matteo's work for AssignmentManager which
> is
> > to
> > > > > make
> > > > > > Master more stable.
> > > > > >
> > > > > > Cheers
> > > > > >
> > > > > > On Fri, Sep 23, 2016 at 5:32 AM, 张铎 
> wrote:
> > > > > >
> > > > > > > No, not your fault, at lease, not this time:)
> > > > > > >
> > > > > > > Why I call the code ugly? Can you simply tell me the sequence
> of
> > > > c

[jira] [Created] (HBASE-16702) TestBlockEvictionFromClient is broken

[Heng Chen (JIRA)]

Heng Chen created HBASE-16702:
-

 Summary: TestBlockEvictionFromClient is broken
 Key: HBASE-16702
 URL: https://issues.apache.org/jira/browse/HBASE-16702
 Project: HBase
  Issue Type: Bug
Reporter: Heng Chen






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-16702) TestBlockEvictionFromClient is broken

[Duo Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-16702.
---
Resolution: Duplicate

Duplicated with HBASE-16696.

> TestBlockEvictionFromClient is broken
> -
>
> Key: HBASE-16702
> URL: https://issues.apache.org/jira/browse/HBASE-16702
> Project: HBase
>  Issue Type: Bug
>Reporter: Heng Chen
>
> https://builds.apache.org/job/PreCommit-HBASE-Build/3682/artifact/patchprocess/patch-unit-hbase-server.txt



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)