Re: [ANNOUNCE] New HBase Committer Josh Elser

[张铎]

Congratulations!

2016-12-12 9:03 GMT+08:00 Jerry He :

> Congratulations , Josh!
>
> Good work on the PQS too.
>
> Jerry
>
> On Sun, Dec 11, 2016 at 12:14 PM, Josh Elser  wrote:
>
> > Thanks, all. I'm looking forward to continuing to work with you all!
> >
> >
> > Nick Dimiduk wrote:
> >
> >> On behalf of the Apache HBase PMC, I am pleased to announce that Josh
> >> Elser
> >> has accepted the PMC's invitation to become a committer on the project.
> We
> >> appreciate all of Josh's generous contributions thus far and look
> forward
> >> to his continued involvement.
> >>
> >> Allow me to be the first to congratulate and welcome Josh into his new
> >> role!
> >>
> >>
>

Re: protobuf 2.5 and 3.1

[张铎]

In master we use protobuf-maven-plugin to run protoc. The plugin will
download protoc binary from the maven repo so you do not need to install
protoc on your machine manually.

So you can just install pb 2.5 on your machine.

For the general question that how to have 2 protobuf versions on one
machine, I think you need to build protoc from source and install it
manually, and use symbol link to select the prefered version before you run
mvn command.

Thanks.

2016-12-09 16:00 GMT+08:00 Stephen Jiang :

> master branch is in protobuf version 3.1; branch-1.1 is in protobuf version
> 2.5
>
> I upgraded to version 3.1 in my machine so that I can work on master
> branch; now I need to run the protobuf to generate file in branch-1.1 and
> get the error:
>
> *[ERROR] Failed to execute goal
> org.apache.hadoop:hadoop-maven-plugins:2.5.1:protoc (compile-protoc) on
> project hbase-protocol: org.apache.maven.plugin.MojoExecutionException:
> protoc version is 'libprotoc 3.1.0', expected version is '2.5.0' -> [Help
> 1]*
>
> Is there a easy way to have 2 protobuf version side-by-side and specify in
> mvn command?
>
> Thanks
> Stephen
>

Re: meta region written by 2.0 cannot be opened by 1.x

[张铎]

I think the imcompatible change is HBASE-15265? HBASE-14949 aims to fix the
incompatible... Just like HBASE-16189 to HBASE-10800.

I think we should mark HBASE-15265 and HBASE-10800 as Incompatible Change?

2016-12-07 18:29 GMT+08:00 Ted Yu :

> Shouldn't HBASE-14949 be marked Incompatible Change ?
>
> On Wed, Dec 7, 2016 at 2:17 AM, 张铎  wrote:
>
> > See this one
> >
> > https://issues.apache.org/jira/browse/HBASE-14949
> >
> > 2016-12-07 18:09 GMT+08:00 Ted Yu :
> >
> > > Which other features ?
> > > Can you name them ?
> > >
> > > Thanks
> > >
> > > On Wed, Dec 7, 2016 at 2:06 AM, 张铎  wrote:
> > >
> > > > I believe there are other features that require upgrading to a
> specific
> > > > version in each 1.x release before rolling upgrading to 2.0? I think
> > this
> > > > is a typical trick to add incompatible changes...
> > > >
> > > > 2016-12-07 18:02 GMT+08:00 ramkrishna vasudevan <
> > > > ramkrishna.s.vasude...@gmail.com>:
> > > >
> > > > > If its going to be a pain then we need to support writing the older
> > > class
> > > > > name in the FFT (trailer) and then use our internal mapping.
> > > > >
> > > > > On Wed, Dec 7, 2016 at 3:22 PM, Anoop John 
> > > > wrote:
> > > > >
> > > > > > I read the comments in the issue 16189..  This concern was raised
> > > > > > there already..   This is what I replied there.
> > > > > > "Ya we may need to make such a need. User on 1.x has to first
> > upgrade
> > > > > > (rolling upgrade supported) to latest version in 1.x and then to
> > 2.0.
> > > > > > This was discussed some other place also. I believe in the mail
> > > thread
> > > > > > where we discussed abt rolling upgrade support in 2.0"
> > > > > >
> > > > > > Not able to find the original mail thread which discussed abt
> > rolling
> > > > > > upgrade support in 2.0Pls correct if took some thing wrong
> > way..
> > > > > > Ya I agree that this way of indirection might be inconvenience
> for
> > > > > > users.
> > > > > >
> > > > > >
> > > > > > -Anoop-
> > > > > >
> > > > > > On Wed, Dec 7, 2016 at 3:13 PM, Ted Yu 
> > wrote:
> > > > > > > bq. write old name only and within code we will have to map it
> > with
> > > > new
> > > > > > > name.
> > > > > > >
> > > > > > > This is a viable approach.
> > > > > > > Even if we document upgrading to 1.2.3+, some users may not
> > perform
> > > > > this
> > > > > > > step before upgrading to 2.0
> > > > > > >
> > > > > > > On Tue, Dec 6, 2016 at 10:04 PM, Anoop John <
> > anoop.hb...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > >> Ya that bug raised by Enis was very valid..  So this means
> when
> > > > > > >> rolling upgrade happens, if some of the other RSs with version
> > > > <1.2.3
> > > > > > >> , which is not having this fix,  this issue might come up!
> > > > > > >> How to address then?   Do we need to enforce a 1.2.3+ upgrade
> > 1st
> > > > and
> > > > > > >> then only 2.0 rolling upgrade?
> > > > > > >>
> > > > > > >> Or else we will need to fix in 2.0..  We write the new
> > Comparator
> > > > > > >> class name in FFT (trunk code)   To fix, we might have to
> write
> > > old
> > > > > > >> name only and within code we will have to map it with new
> name.
> > It
> > > > > > >> will be ugly! It can be fixed only in 3.0 may be..  But that
> can
> > > > make
> > > > > > >> the rolling upgrade story easier for users..   Just saying the
> > > > > > >> possibilities..
> > > > > > >>
> > > > > > >> Thanks Ted to bring it up again.
> > > > > > >>
> > > > > > >> -Anoop-
> > > > > > >>
> > > > > > >> On Wed, Dec 7, 2016 at 10:04 AM, ramkr

Re: meta region written by 2.0 cannot be opened by 1.x

[张铎]

And a bad news is that this patch is not contained in 1.2.x and 1.1.x which
means it is not safe to rolling upgrade from 1.2.x or 1.1.x to 2.0 if we
change the default wal implementation to AsyncFSWA, or a user set the wal
implementation to asyncfs manually...

2016-12-07 18:17 GMT+08:00 张铎 :

> See this one
>
> https://issues.apache.org/jira/browse/HBASE-14949
>
> 2016-12-07 18:09 GMT+08:00 Ted Yu :
>
>> Which other features ?
>> Can you name them ?
>>
>> Thanks
>>
>> On Wed, Dec 7, 2016 at 2:06 AM, 张铎  wrote:
>>
>> > I believe there are other features that require upgrading to a specific
>> > version in each 1.x release before rolling upgrading to 2.0? I think
>> this
>> > is a typical trick to add incompatible changes...
>> >
>> > 2016-12-07 18:02 GMT+08:00 ramkrishna vasudevan <
>> > ramkrishna.s.vasude...@gmail.com>:
>> >
>> > > If its going to be a pain then we need to support writing the older
>> class
>> > > name in the FFT (trailer) and then use our internal mapping.
>> > >
>> > > On Wed, Dec 7, 2016 at 3:22 PM, Anoop John 
>> > wrote:
>> > >
>> > > > I read the comments in the issue 16189..  This concern was raised
>> > > > there already..   This is what I replied there.
>> > > > "Ya we may need to make such a need. User on 1.x has to first
>> upgrade
>> > > > (rolling upgrade supported) to latest version in 1.x and then to
>> 2.0.
>> > > > This was discussed some other place also. I believe in the mail
>> thread
>> > > > where we discussed abt rolling upgrade support in 2.0"
>> > > >
>> > > > Not able to find the original mail thread which discussed abt
>> rolling
>> > > > upgrade support in 2.0Pls correct if took some thing wrong way..
>> > > > Ya I agree that this way of indirection might be inconvenience for
>> > > > users.
>> > > >
>> > > >
>> > > > -Anoop-
>> > > >
>> > > > On Wed, Dec 7, 2016 at 3:13 PM, Ted Yu  wrote:
>> > > > > bq. write old name only and within code we will have to map it
>> with
>> > new
>> > > > > name.
>> > > > >
>> > > > > This is a viable approach.
>> > > > > Even if we document upgrading to 1.2.3+, some users may not
>> perform
>> > > this
>> > > > > step before upgrading to 2.0
>> > > > >
>> > > > > On Tue, Dec 6, 2016 at 10:04 PM, Anoop John <
>> anoop.hb...@gmail.com>
>> > > > wrote:
>> > > > >
>> > > > >> Ya that bug raised by Enis was very valid..  So this means when
>> > > > >> rolling upgrade happens, if some of the other RSs with version
>> > <1.2.3
>> > > > >> , which is not having this fix,  this issue might come up!
>> > > > >> How to address then?   Do we need to enforce a 1.2.3+ upgrade 1st
>> > and
>> > > > >> then only 2.0 rolling upgrade?
>> > > > >>
>> > > > >> Or else we will need to fix in 2.0..  We write the new Comparator
>> > > > >> class name in FFT (trunk code)   To fix, we might have to write
>> old
>> > > > >> name only and within code we will have to map it with new name.
>> It
>> > > > >> will be ugly! It can be fixed only in 3.0 may be..  But that can
>> > make
>> > > > >> the rolling upgrade story easier for users..   Just saying the
>> > > > >> possibilities..
>> > > > >>
>> > > > >> Thanks Ted to bring it up again.
>> > > > >>
>> > > > >> -Anoop-
>> > > > >>
>> > > > >> On Wed, Dec 7, 2016 at 10:04 AM, ramkrishna vasudevan
>> > > > >>  wrote:
>> > > > >> > I think when Enis reported the issue (
>> > > > >> > https://issues.apache.org/jira/browse/HBASE-16189), in rolling
>> > > > upgrade
>> > > > >> > regions could move around. So a region created in 2.0 moved to
>> a
>> > > > server
>> > > > >> > with 1.x.
>> > > > >> >
>> > > > >> > Regards
>> > > > >> > Ram
>&

Re: meta region written by 2.0 cannot be opened by 1.x

[张铎]

See this one

https://issues.apache.org/jira/browse/HBASE-14949

2016-12-07 18:09 GMT+08:00 Ted Yu :

> Which other features ?
> Can you name them ?
>
> Thanks
>
> On Wed, Dec 7, 2016 at 2:06 AM, 张铎  wrote:
>
> > I believe there are other features that require upgrading to a specific
> > version in each 1.x release before rolling upgrading to 2.0? I think this
> > is a typical trick to add incompatible changes...
> >
> > 2016-12-07 18:02 GMT+08:00 ramkrishna vasudevan <
> > ramkrishna.s.vasude...@gmail.com>:
> >
> > > If its going to be a pain then we need to support writing the older
> class
> > > name in the FFT (trailer) and then use our internal mapping.
> > >
> > > On Wed, Dec 7, 2016 at 3:22 PM, Anoop John 
> > wrote:
> > >
> > > > I read the comments in the issue 16189..  This concern was raised
> > > > there already..   This is what I replied there.
> > > > "Ya we may need to make such a need. User on 1.x has to first upgrade
> > > > (rolling upgrade supported) to latest version in 1.x and then to 2.0.
> > > > This was discussed some other place also. I believe in the mail
> thread
> > > > where we discussed abt rolling upgrade support in 2.0"
> > > >
> > > > Not able to find the original mail thread which discussed abt rolling
> > > > upgrade support in 2.0Pls correct if took some thing wrong way..
> > > > Ya I agree that this way of indirection might be inconvenience for
> > > > users.
> > > >
> > > >
> > > > -Anoop-
> > > >
> > > > On Wed, Dec 7, 2016 at 3:13 PM, Ted Yu  wrote:
> > > > > bq. write old name only and within code we will have to map it with
> > new
> > > > > name.
> > > > >
> > > > > This is a viable approach.
> > > > > Even if we document upgrading to 1.2.3+, some users may not perform
> > > this
> > > > > step before upgrading to 2.0
> > > > >
> > > > > On Tue, Dec 6, 2016 at 10:04 PM, Anoop John  >
> > > > wrote:
> > > > >
> > > > >> Ya that bug raised by Enis was very valid..  So this means when
> > > > >> rolling upgrade happens, if some of the other RSs with version
> > <1.2.3
> > > > >> , which is not having this fix,  this issue might come up!
> > > > >> How to address then?   Do we need to enforce a 1.2.3+ upgrade 1st
> > and
> > > > >> then only 2.0 rolling upgrade?
> > > > >>
> > > > >> Or else we will need to fix in 2.0..  We write the new Comparator
> > > > >> class name in FFT (trunk code)   To fix, we might have to write
> old
> > > > >> name only and within code we will have to map it with new name. It
> > > > >> will be ugly! It can be fixed only in 3.0 may be..  But that can
> > make
> > > > >> the rolling upgrade story easier for users..   Just saying the
> > > > >> possibilities..
> > > > >>
> > > > >> Thanks Ted to bring it up again.
> > > > >>
> > > > >> -Anoop-
> > > > >>
> > > > >> On Wed, Dec 7, 2016 at 10:04 AM, ramkrishna vasudevan
> > > > >>  wrote:
> > > > >> > I think when Enis reported the issue (
> > > > >> > https://issues.apache.org/jira/browse/HBASE-16189), in rolling
> > > > upgrade
> > > > >> > regions could move around. So a region created in 2.0 moved to a
> > > > server
> > > > >> > with 1.x.
> > > > >> >
> > > > >> > Regards
> > > > >> > Ram
> > > > >> >
> > > > >> >
> > > > >> > On Wed, Dec 7, 2016 at 1:27 AM, Stack  wrote:
> > > > >> >
> > > > >> >> On Tue, Dec 6, 2016 at 10:19 AM, Ted Yu 
> > > wrote:
> > > > >> >>
> > > > >> >> > Looking at http://hbase.apache.org/book.
> > html#executing.the.0.96.
> > > > >> upgrade
> > > > >> >> ,
> > > > >> >> > there is step of running "bin/hbase upgrade -check"
> > > > >> >> >
> > > > >> >> > How about adding a sample hfile which contains
> > > > >> &g

Re: meta region written by 2.0 cannot be opened by 1.x

[张铎]

I believe there are other features that require upgrading to a specific
version in each 1.x release before rolling upgrading to 2.0? I think this
is a typical trick to add incompatible changes...

2016-12-07 18:02 GMT+08:00 ramkrishna vasudevan <
ramkrishna.s.vasude...@gmail.com>:

> If its going to be a pain then we need to support writing the older class
> name in the FFT (trailer) and then use our internal mapping.
>
> On Wed, Dec 7, 2016 at 3:22 PM, Anoop John  wrote:
>
> > I read the comments in the issue 16189..  This concern was raised
> > there already..   This is what I replied there.
> > "Ya we may need to make such a need. User on 1.x has to first upgrade
> > (rolling upgrade supported) to latest version in 1.x and then to 2.0.
> > This was discussed some other place also. I believe in the mail thread
> > where we discussed abt rolling upgrade support in 2.0"
> >
> > Not able to find the original mail thread which discussed abt rolling
> > upgrade support in 2.0Pls correct if took some thing wrong way..
> > Ya I agree that this way of indirection might be inconvenience for
> > users.
> >
> >
> > -Anoop-
> >
> > On Wed, Dec 7, 2016 at 3:13 PM, Ted Yu  wrote:
> > > bq. write old name only and within code we will have to map it with new
> > > name.
> > >
> > > This is a viable approach.
> > > Even if we document upgrading to 1.2.3+, some users may not perform
> this
> > > step before upgrading to 2.0
> > >
> > > On Tue, Dec 6, 2016 at 10:04 PM, Anoop John 
> > wrote:
> > >
> > >> Ya that bug raised by Enis was very valid..  So this means when
> > >> rolling upgrade happens, if some of the other RSs with version <1.2.3
> > >> , which is not having this fix,  this issue might come up!
> > >> How to address then?   Do we need to enforce a 1.2.3+ upgrade 1st and
> > >> then only 2.0 rolling upgrade?
> > >>
> > >> Or else we will need to fix in 2.0..  We write the new Comparator
> > >> class name in FFT (trunk code)   To fix, we might have to write old
> > >> name only and within code we will have to map it with new name. It
> > >> will be ugly! It can be fixed only in 3.0 may be..  But that can make
> > >> the rolling upgrade story easier for users..   Just saying the
> > >> possibilities..
> > >>
> > >> Thanks Ted to bring it up again.
> > >>
> > >> -Anoop-
> > >>
> > >> On Wed, Dec 7, 2016 at 10:04 AM, ramkrishna vasudevan
> > >>  wrote:
> > >> > I think when Enis reported the issue (
> > >> > https://issues.apache.org/jira/browse/HBASE-16189), in rolling
> > upgrade
> > >> > regions could move around. So a region created in 2.0 moved to a
> > server
> > >> > with 1.x.
> > >> >
> > >> > Regards
> > >> > Ram
> > >> >
> > >> >
> > >> > On Wed, Dec 7, 2016 at 1:27 AM, Stack  wrote:
> > >> >
> > >> >> On Tue, Dec 6, 2016 at 10:19 AM, Ted Yu 
> wrote:
> > >> >>
> > >> >> > Looking at http://hbase.apache.org/book.html#executing.the.0.96.
> > >> upgrade
> > >> >> ,
> > >> >> > there is step of running "bin/hbase upgrade -check"
> > >> >> >
> > >> >> > How about adding a sample hfile which contains
> > >> >> > CellComparator$MetaCellComparator
> > >> >> > so that the upgrade check can read and verify ?
> > >> >> >
> > >> >> >
> > >> >>
> > >> >> We don't support downgrade. Never have.
> > >> >> St.Ack
> > >> >>
> > >> >>
> > >> >>
> > >> >> > On Tue, Dec 6, 2016 at 8:50 AM, Ted Yu 
> > wrote:
> > >> >> >
> > >> >> > > The build I used yesterday didn't include HBASE-16189
> > >> >> > > <https://issues.apache.org/jira/browse/HBASE-16189>
> > >> >> > >
> > >> >> > > Once it is included, the cluster can be downgraded fine.
> > >> >> > >
> > >> >> > > I wonder how users would know that their existing deployment
> has
> > >> >> > > HBASE-16189 <https://issues.apache.org/jira/browse/HBASE-16189
> >
> > &

Re: meta region written by 2.0 cannot be opened by 1.x

[张铎]

Could this be solved by hosting meta only on master?

BTW, MetaCellComparator is introduced in HBASE-10800.

Thanks.

2016-12-06 17:44 GMT+08:00 Ted Yu :

> Hi,
> When I restarted a cluster with 1.1 , I found that hbase:meta region
> (written to by the previously deployed 2.0) couldn't be opened:
>
> Caused by: java.io.IOException:
> org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
> HFile Trailer from file hdfs://yz1.xx.com:8020/apps/  hbase/data/data/
>  hbase/meta/1588230740/info/599fc8a37311414e876803312009a986
> at
> org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(HStore.java:
> 579)
> at
> org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:
> 534)
> at
> org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:275)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.
> java:5150)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:912)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:909)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
> reading HFile Trailer from file hdfs://
> yz1.xx.com:8020/apps/hbase/data/data/hbase/ meta/1588230740/
>  info/599fc8a37311414e876803312009a986
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:483)
> at
> org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:511)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.
> (StoreFile.java:1128)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.
> open(StoreFileInfo.java:267)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:409)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.
> createReader(StoreFile.java:517)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(
> HStore.java:687)
> at
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:130)
> at
> org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:554)
> at
> org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:551)
> ... 6 more
> Caused by: java.io.IOException: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.CellComparator$MetaCellComparator
> at
> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.getComparatorClass(
> FixedFileTrailer.java:581)
> at
> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.deserializeFromPB(
> FixedFileTrailer.java:300)
> at
> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.
> deserialize(FixedFileTrailer.java:242)
> at
> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(
> FixedFileTrailer.java:407)
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:468)
> ... 15 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.CellComparator$MetaCellComparator
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at
> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.getComparatorClass(
> FixedFileTrailer.java:579)
>
> When user does rolling upgrade from 1.1 to 2.0, the above may cause problem
> if hbase:meta region is updated by server running 2.0 but later assigned to
> a region server which still runs 1.1 (due to crash of the server running
> 2.0, e.g.)
>
> I want to get community feedback on the severity of this issue.
>
> Thanks
>

Re: mini cluster not starting due to unhandled com.lmax.disruptor.dsl.Disruptor. (eclipse only, not in mvn)

[张铎]

Try mvn eclispe:eclipse again? We upgraded disruptor to 3.3.6 recently.

2016-11-28 10:48 GMT+08:00 Stephen Jiang :

> I had problem to start mini cluster in eclipse in the last few days.  In
> master branch (without any additional change), I got the following FATAL
> error (running the UT using maven has NO problem):
>
> 2016-11-27 18:44:58,102 FATAL [M:0;10.10.0.153:59578]
> master.HMaster(2241):
> Master server abort: loaded coprocessors are: []
> 2016-11-27 18:44:58,102 FATAL [M:0;10.10.0.153:59578]
> master.HMaster(2244):
> Unhandled:
> com.lmax.disruptor.dsl.Disruptor.(Lcom/lmax/
> disruptor/EventFactory;ILjava/util/concurrent/ThreadFactory;
> Lcom/lmax/disruptor/dsl/ProducerType;Lcom/lmax/disruptor/WaitStrategy;)V
> java.lang.NoSuchMethodError:
> com.lmax.disruptor.dsl.Disruptor.(Lcom/lmax/
> disruptor/EventFactory;ILjava/util/concurrent/ThreadFactory;
> Lcom/lmax/disruptor/dsl/ProducerType;Lcom/lmax/disruptor/WaitStrategy;)V
> at org.apache.hadoop.hbase.regionserver.wal.FSHLog.(FSHLog.java:230)
> at
> org.apache.hadoop.hbase.wal.FSHLogProvider.createWAL(
> FSHLogProvider.java:80)
> at
> org.apache.hadoop.hbase.wal.FSHLogProvider.createWAL(
> FSHLogProvider.java:39)
> at
> org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(
> AbstractFSWALProvider.java:132)
> at
> org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(
> AbstractFSWALProvider.java:52)
> at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:242)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.
> getWAL(HRegionServer.java:1929)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.
> buildServerLoad(HRegionServer.java:1247)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(
> HRegionServer.java:1205)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.
> run(HRegionServer.java:1022)
> at java.lang.Thread.run(Thread.java:745)
>

Re: Use experience and performance data of offheap from Alibaba online cluster

[张铎]

I can not see the images either...

Du, Jingcheng 于2016年11月18日 周五16:57写道：

> Thanks Yu for the sharing, great achievements.
> It seems the images cannot be displayed? Maybe just me?
>
> Regards,
> Jingcheng
>
> From: Yu Li [mailto:car...@gmail.com]
> Sent: Friday, November 18, 2016 4:11 PM
> To: u...@hbase.apache.org; dev@hbase.apache.org
> Subject: Use experience and performance data of offheap from Alibaba
> online cluster
>
> Dear all,
>
> We have backported read path offheap (HBASE-11425) to our customized
> hbase-1.1.2 (thanks @Anoop for the help/support) and run it online for more
> than a month, and would like to share our experience, for what it's worth
> (smile).
>
> Generally speaking, we gained a better and more stable
> throughput/performance with offheap, and below are some details:
>
> 1. QPS become more stable with offheap
>
> Performance w/o offheap:
>
> [cid:part1.582d4b6424f071c]
>
> Performance w/ offheap:
>
> [cid:part2.582d4b6424f071c]
>
> These data come from our online A/B test cluster (with 450 physical
> machines, and each with 256G memory + 64 core) with real world workloads,
> it shows using offheap we could gain a more stable throughput as well as
> better performance
>
> Not showing fully online data here because for online we published the
> version with both offheap and NettyRpcServer together, so no standalone
> comparison data for offheap
>
> 2. Full GC frequency and cost
>
> Average Full GC STW time reduce from 11s to 7s with offheap.
>
> 3. Young GC frequency and cost
>
> No performance degradation observed with offheap.
>
> 4. Peak throughput of one single RS
>
> On Singles Day (11/11), peak throughput of one single RS reached 100K,
> among which 90K from Get. Plus internet in/out data we could know the
> average result size of get request is ~1KB
>
> [cid:part3.582d4b6424f071c]
>
> Offheap are used on all online machines (more than 1600 nodes) instead of
> LruCache, so the above QPS is gained from offheap bucketcache, along with
> NettyRpcServer(HBASE-15756).
> Just let us know if any comments. Thanks.
>
> Best Regards,
> Yu
>
>
>
>
>
>
>

Re: DISCUSS: Strip xref from our site generation

[张铎]

+1.

2016-11-17 15:57 GMT+08:00 Nick Dimiduk :

> Agree the value add is minimal.
>
> On Wed, Nov 16, 2016 at 9:41 PM Stack  wrote:
>
> > The xref pages were useful once before grepcode and easy navigation in
> > apache git and github. Now, not so much. It is a long time since I've
> seen
> > anyone make a citation using xref pointers. Meantime the pages add a
> bunch
> > of bulk to our site.
> >
> > I suggest we undo their generation as part of site build and remove the
> > xref links from our site menus?
> >
> > Thoughts?
> >
> > S
> >
>

Re: Please update your jenkins job configs ASAP

[张铎]

https://builds.apache.org/job/PreCommit-HBASE-Build/4428/consoleFull

This one was executed on H18 but still failed because of OOM, so I guess
the problem is not H2?

2016-11-12 2:37 GMT+08:00 Stack :

> Thanks Gavin. The not-h2 label was added because our jobs were failing when
> they ran on h2.
>
> Our jobs have just started failing w/ OOMEs (HBASE-17074).  You seeing that
> anywhere else and if so, any input before we start trying to untangle (I
> ask you because you have more insight than we do)?
>
> Thanks,
> S
>
>
> On Thu, Nov 10, 2016 at 7:44 PM, Gavin McDonald 
> wrote:
>
> > Hi HBasers!
> >
> > A couple of days ago I removed labels from the jenkins nodes (as promised
> > in an earlier mail) and at the same
> > time I changed your builds to remove the ‘yahoo-no-h2r’ label and replace
> > it with ‘Hadoop’.
> >
> > Looks like your jobs have reverted back to the ‘ubuntu-not-h2’ label and
> > are now awaiting in the queue for a node that
> > doesn’t exist.
> >
> > Please update your jobs using whatever method (job-dsl etc) so your jobs
> > can resume building.
> >
> > See https://cwiki.apache.org/confluence/display/INFRA/
> Jenkins+node+labels
> > 
> > for the upto date slimmed down
> > label listing. As you’ll see the ‘Hadoop’ label gives you (currently a
> > minimum of) 10 nodes and choosing
> > (Hadoop||ubuntu) gives you 27 nodes!.
> >
> > Recap - there is NO YAHOO-NOT-H2 LABEL. It is not needed - H2 is the same
> > as any other node.
> >
> > Thanks!
> >
> > Gav… (ASF Infra.)
>

Re: [DISCUSS] Drop legacy hadoop support at the 2.0 release

[张铎]

See HBASE-16887

2016-10-20 22:50 GMT+08:00 Sean Busbey :

> Unfortunately, I think this means we'll need to update the hadoopcheck
> versions to vary by branch.
>
> On Wed, Oct 19, 2016 at 6:21 PM, 张铎  wrote:
> > OK, seems no objection. I will file a issue to modify the hadoop version
> > support matrix.
> >
> > And I think we also need to change the hadoopcheck versions for our
> > precommit check?
> >
> > Thanks all.
> >
> > 2016-10-20 1:14 GMT+08:00 Andrew Purtell :
> >
> >> FWIW, we are running 2.7.x in production and it's stable.
> >>
> >>
> >> On Tue, Oct 18, 2016 at 10:18 PM, Sean Busbey 
> wrote:
> >>
> >> > we had not decided yet AFAIK. a big concern was the lack of
> >> > maintenance releases on more recent Hadoop versions and the perception
> >> > that 2.4 and 2.5 were the last big stable release lines.
> >> >
> >> > 2.6 and 2.7 have both gotten a few maintenance releases now, so maybe
> >> > this isn't a concern any more?
> >> >
> >> > On Tue, Oct 18, 2016 at 7:00 PM, Enis Söztutar 
> >> wrote:
> >> > > I thought we already decided to do that, no?
> >> > >
> >> > > Enis
> >> > >
> >> > > On Tue, Oct 18, 2016 at 6:56 PM, Ted Yu 
> wrote:
> >> > >
> >> > >> Looking at http://hadoop.apache.org/releases.html , 2.5.x hasn't
> got
> >> > new
> >> > >> release for almost two years.
> >> > >>
> >> > >> Seems fine to drop support for 2.4 and 2.5
> >> > >>
> >> > >> On Tue, Oct 18, 2016 at 6:42 PM, Duo Zhang 
> >> wrote:
> >> > >>
> >> > >> > This is the current hadoop version support matrix
> >> > >> >
> >> > >> > https://hbase.apache.org/book.html#hadoop
> >> > >> >
> >> > >> > 2016-10-19 9:40 GMT+08:00 Duo Zhang :
> >> > >> >
> >> > >> > > To be specific, hadoop-2.4.x and hadoop-2.5.x.
> >> > >> > >
> >> > >> > > The latest releases for these two lines are about two years
> ago,
> >> so
> >> > I
> >> > >> > > think it is the time to drop the support of them when 2.0 out.
> >> Then
> >> > we
> >> > >> > > could drop some code in our hadoop-compat module as we may
> need to
> >> > add
> >> > >> > some
> >> > >> > > code for the incoming hadoop-3.0...
> >> > >> > >
> >> > >> > > Thanks.
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
> >>
> >>
> >> --
> >> Best regards,
> >>
> >>- Andy
> >>
> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> >> (via Tom White)
> >>
>
>
>
> --
> busbey
>

Re: [DISCUSS] Drop legacy hadoop support at the 2.0 release

[张铎]

OK, seems no objection. I will file a issue to modify the hadoop version
support matrix.

And I think we also need to change the hadoopcheck versions for our
precommit check?

Thanks all.

2016-10-20 1:14 GMT+08:00 Andrew Purtell :

> FWIW, we are running 2.7.x in production and it's stable.
>
>
> On Tue, Oct 18, 2016 at 10:18 PM, Sean Busbey  wrote:
>
> > we had not decided yet AFAIK. a big concern was the lack of
> > maintenance releases on more recent Hadoop versions and the perception
> > that 2.4 and 2.5 were the last big stable release lines.
> >
> > 2.6 and 2.7 have both gotten a few maintenance releases now, so maybe
> > this isn't a concern any more?
> >
> > On Tue, Oct 18, 2016 at 7:00 PM, Enis Söztutar 
> wrote:
> > > I thought we already decided to do that, no?
> > >
> > > Enis
> > >
> > > On Tue, Oct 18, 2016 at 6:56 PM, Ted Yu  wrote:
> > >
> > >> Looking at http://hadoop.apache.org/releases.html , 2.5.x hasn't got
> > new
> > >> release for almost two years.
> > >>
> > >> Seems fine to drop support for 2.4 and 2.5
> > >>
> > >> On Tue, Oct 18, 2016 at 6:42 PM, Duo Zhang 
> wrote:
> > >>
> > >> > This is the current hadoop version support matrix
> > >> >
> > >> > https://hbase.apache.org/book.html#hadoop
> > >> >
> > >> > 2016-10-19 9:40 GMT+08:00 Duo Zhang :
> > >> >
> > >> > > To be specific, hadoop-2.4.x and hadoop-2.5.x.
> > >> > >
> > >> > > The latest releases for these two lines are about two years ago,
> so
> > I
> > >> > > think it is the time to drop the support of them when 2.0 out.
> Then
> > we
> > >> > > could drop some code in our hadoop-compat module as we may need to
> > add
> > >> > some
> > >> > > code for the incoming hadoop-3.0...
> > >> > >
> > >> > > Thanks.
> > >> > >
> > >> >
> > >>
> >
>
>
>
> --
> Best regards,
>
>- Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

Re: [ANNOUNCE] Stephen Yuan Jiang joins Apache HBase PMC

[张铎]

Congratulations!

2016-10-17 9:07 GMT+08:00 Heng Chen :

> Congrats!  :)
>
> 2016-10-16 8:19 GMT+08:00 Jerry He :
> > Congratulations, Stephen.
> >
> > Jerry
> >
> > On Fri, Oct 14, 2016 at 12:56 PM, Dima Spivak 
> wrote:
> >
> >> Congrats, Stephen!
> >>
> >> -Dima
> >>
> >> On Fri, Oct 14, 2016 at 11:27 AM, Enis Söztutar 
> wrote:
> >>
> >> > On behalf of the Apache HBase PMC, I am happy to announce that Stephen
> >> has
> >> > accepted our invitation to become a PMC member of the Apache HBase
> >> project.
> >> >
> >> > Stephen has been working on HBase for a couple of years, and is
> already a
> >> > committer for more than a year. Apart from his contributions in proc
> v2,
> >> > hbck and other areas, he is also helping for the 2.0 release which is
> the
> >> > most important milestone for the project this year.
> >> >
> >> > Welcome to the PMC Stephen,
> >> > Enis
> >> >
> >>
>

Re: Still Failing: HBase Generate Website

[张铎]

You can setup a branch with that commit in I think.

Maybe we can setup a new jenkins job that only executes on H0 and H1?

2016-10-11 9:37 GMT+08:00 Ted Yu :

> Adding '-Xnative.verbose=true' can be done - but the commit has been
> reverted, not sure if the output is useful.
>
> To my knowledge, nobody has access to H0 (or any other Jenkins machines)
> other than Apache admins.
>
> On Mon, Oct 10, 2016 at 6:32 PM, 张铎  wrote:
>
> > Could the infra team give us a temporary permission to login H0 and run
> > some builds manually?
> >
> > https://github.com/jruby/jruby/issues/2913
> >
> > Here they suggest adding -Xnative.verbose=true when building to see the
> > root cause. We could also do this.
> >
> > 2016-10-11 4:24 GMT+08:00 Ted Yu :
> >
> > > For those who are not watching HBASE-16750. We had successful build
> with
> > > the following Ubuntu OS.
> > >
> > > Saravanan and I will keep working on this issue till resolution.
> > >
> > > Thanks to Dima who kept an eye on website builds.
> > >
> > > jenkins@u12-slave-4c6292b0-1:~/workspace/build-support$ uname -a
> > > Linux u12-slave-4c6292b0-1 3.2.0-74-virtual #109-Ubuntu SMP Tue Dec 9
> > > 17:04:48 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
> > > JAVA: /grid/0/jenkins/tools/jdk7/jdk1.7.0_21
> > > Maven: /grid/0/jenkins/tools/maven/apache-maven-3.3.9/bin/mvn
> > >
> > >
> > > On Mon, Oct 10, 2016 at 8:01 AM, Dima Spivak 
> > > wrote:
> > >
> > > > I'll revert and rerun on the same build machine. Whether the problem
> is
> > > > with the environment or not, I don't recall it ever happening before
> > this
> > > > commit and this kind of regression is unacceptable.
> > > >
> > > > On Sunday, October 9, 2016, stack  wrote:
> > > >
> > > > > Did you do the revert meantime Ted?
> > > > >
> > > > > On Oct 8, 2016 19:01, "Ted Yu"  >
> > > > wrote:
> > > > >
> > > > > > Logged INFRA-12731
> > > > > >
> > > > > > On Sat, Oct 8, 2016 at 6:56 PM, 张铎  > > > >
> > > > > wrote:
> > > > > >
> > > > > > > Build environment issue IS a issue. If we do not consider H0
> and
> > H1
> > > > as
> > > > > > > broken, we need to fix the problem.
> > > > > > >
> > > > > > > And we are not sure if all other machines are OK right? As for
> > now,
> > > > the
> > > > > > > failed machines wins, 2 vs. 1.
> > > > > > >
> > > > > > > Thanks.
> > > > > > >
> > > > > > > 2016-10-09 9:36 GMT+08:00 Ted Yu  > > > >:
> > > > > > >
> > > > > > > > This seems to be build environment issue.
> > > > > > > >
> > > > > > > > Build #368 (green) was on H4 while #369 was on H0. #370 was
> on
> > > H1.
> > > > > > > >
> > > > > > > > On Sat, Oct 8, 2016 at 6:32 PM, 张铎  > > > > > wrote:
> > > > > > > >
> > > > > > > > > This maybe related to a jruby issue.
> > > > > > > > >
> > > > > > > > > https://github.com/jruby/jruby/issues/2913
> > > > > > > > >
> > > > > > > > > It depends on the environment thus it may fail on some
> > machines
> > > > > while
> > > > > > > > > success on some other machines.
> > > > > > > > >
> > > > > > > > > 2016-10-09 9:29 GMT+08:00 Ted Yu  > > > > >:
> > > > > > > > >
> > > > > > > > > > This was the build 16750 went into:
> > > > > > > > > >
> > > > > > > > > > https://builds.apache.org/job/
> hbase_generate_website/368/
> > > > > > > > > >
> > > > > > > > > > which was green.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Sat, Oct 8, 2016 at 6:10 PM, Sean Busbey <
> > > > bus...@cloudera.com
> > > > > >
> > > > > > > > wrote:
> > > > >

Re: Still Failing: HBase Generate Website

[张铎]

Could the infra team give us a temporary permission to login H0 and run
some builds manually?

https://github.com/jruby/jruby/issues/2913

Here they suggest adding -Xnative.verbose=true when building to see the
root cause. We could also do this.

2016-10-11 4:24 GMT+08:00 Ted Yu :

> For those who are not watching HBASE-16750. We had successful build with
> the following Ubuntu OS.
>
> Saravanan and I will keep working on this issue till resolution.
>
> Thanks to Dima who kept an eye on website builds.
>
> jenkins@u12-slave-4c6292b0-1:~/workspace/build-support$ uname -a
> Linux u12-slave-4c6292b0-1 3.2.0-74-virtual #109-Ubuntu SMP Tue Dec 9
> 17:04:48 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
> JAVA: /grid/0/jenkins/tools/jdk7/jdk1.7.0_21
> Maven: /grid/0/jenkins/tools/maven/apache-maven-3.3.9/bin/mvn
>
>
> On Mon, Oct 10, 2016 at 8:01 AM, Dima Spivak 
> wrote:
>
> > I'll revert and rerun on the same build machine. Whether the problem is
> > with the environment or not, I don't recall it ever happening before this
> > commit and this kind of regression is unacceptable.
> >
> > On Sunday, October 9, 2016, stack  wrote:
> >
> > > Did you do the revert meantime Ted?
> > >
> > > On Oct 8, 2016 19:01, "Ted Yu" >
> > wrote:
> > >
> > > > Logged INFRA-12731
> > > >
> > > > On Sat, Oct 8, 2016 at 6:56 PM, 张铎  > >
> > > wrote:
> > > >
> > > > > Build environment issue IS a issue. If we do not consider H0 and H1
> > as
> > > > > broken, we need to fix the problem.
> > > > >
> > > > > And we are not sure if all other machines are OK right? As for now,
> > the
> > > > > failed machines wins, 2 vs. 1.
> > > > >
> > > > > Thanks.
> > > > >
> > > > > 2016-10-09 9:36 GMT+08:00 Ted Yu  > >:
> > > > >
> > > > > > This seems to be build environment issue.
> > > > > >
> > > > > > Build #368 (green) was on H4 while #369 was on H0. #370 was on
> H1.
> > > > > >
> > > > > > On Sat, Oct 8, 2016 at 6:32 PM, 张铎  > > > wrote:
> > > > > >
> > > > > > > This maybe related to a jruby issue.
> > > > > > >
> > > > > > > https://github.com/jruby/jruby/issues/2913
> > > > > > >
> > > > > > > It depends on the environment thus it may fail on some machines
> > > while
> > > > > > > success on some other machines.
> > > > > > >
> > > > > > > 2016-10-09 9:29 GMT+08:00 Ted Yu  > > >:
> > > > > > >
> > > > > > > > This was the build 16750 went into:
> > > > > > > >
> > > > > > > > https://builds.apache.org/job/hbase_generate_website/368/
> > > > > > > >
> > > > > > > > which was green.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sat, Oct 8, 2016 at 6:10 PM, Sean Busbey <
> > bus...@cloudera.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > If the website can't build on the build machine then it
> can't
> > > > > build.
> > > > > > > > >
> > > > > > > > > We've worked hard as a community to automate this stuff
> and I
> > > > don't
> > > > > > > want
> > > > > > > > to
> > > > > > > > > go backwards.
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Sean Busbey
> > > > > > > > >
> > > > > > > > > On Oct 8, 2016 14:48, "Ted Yu"  > > > wrote:
> > > > > > > > >
> > > > > > > > > > I did a local build:
> > > > > > > > > >
> > > > > > > > > > Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c0
> > > 7478323dc5;
> > > > > > > > > > 2015-11-10T16:41:47+00:00)
> > > > > > > > > > Maven home: /apache-maven-3.3.9
> > > > > > > > > > Java version: 1.8.0_101, vendor: Oracle Corporation
> > > > > > &g

Re: Still Failing: HBase Generate Website

[张铎]

Build environment issue IS a issue. If we do not consider H0 and H1 as
broken, we need to fix the problem.

And we are not sure if all other machines are OK right? As for now, the
failed machines wins, 2 vs. 1.

Thanks.

2016-10-09 9:36 GMT+08:00 Ted Yu :

> This seems to be build environment issue.
>
> Build #368 (green) was on H4 while #369 was on H0. #370 was on H1.
>
> On Sat, Oct 8, 2016 at 6:32 PM, 张铎  wrote:
>
> > This maybe related to a jruby issue.
> >
> > https://github.com/jruby/jruby/issues/2913
> >
> > It depends on the environment thus it may fail on some machines while
> > success on some other machines.
> >
> > 2016-10-09 9:29 GMT+08:00 Ted Yu :
> >
> > > This was the build 16750 went into:
> > >
> > > https://builds.apache.org/job/hbase_generate_website/368/
> > >
> > > which was green.
> > >
> > >
> > > On Sat, Oct 8, 2016 at 6:10 PM, Sean Busbey 
> wrote:
> > >
> > > > If the website can't build on the build machine then it can't build.
> > > >
> > > > We've worked hard as a community to automate this stuff and I don't
> > want
> > > to
> > > > go backwards.
> > > >
> > > > --
> > > > Sean Busbey
> > > >
> > > > On Oct 8, 2016 14:48, "Ted Yu"  wrote:
> > > >
> > > > > I did a local build:
> > > > >
> > > > > Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> > > > > 2015-11-10T16:41:47+00:00)
> > > > > Maven home: /apache-maven-3.3.9
> > > > > Java version: 1.8.0_101, vendor: Oracle Corporation
> > > > > Java home: /jdk1.8.0_101/jre
> > > > > Default locale: en_US, platform encoding: UTF-8
> > > > > OS name: "linux", version: "3.10.0-327.28.2.el7.x86_64", arch:
> > "amd64",
> > > > > family: "unix"
> > > > >
> > > > > [INFO] Final Memory: 30M/1928M
> > > > > [INFO]
> > > > > 
> > > 
> > > > > [INFO] Archetype project created in
> > > > > /hbase/hbase-archetypes/hbase-client-project/target/build-
> > > > > archetype/target/generated-sources/archetype
> > > > > [INFO]
> > > > > 
> > > 
> > > > > [INFO] BUILD SUCCESS
> > > > > [INFO]
> > > > > 
> > > 
> > > > > [INFO] Total time: 3.168 s
> > > > >
> > > > > $ ls -l ./target/site/apache_hbase_reference_guide.pdf
> > > > > -rw-rw-r-- 1 hbase hbase 13535029 Oct  8 19:35
> > > > > ./target/site/apache_hbase_reference_guide.pdf
> > > > >
> > > > > I can attach the generated pdf to the JIRA if you want. It contains
> > the
> > > > new
> > > > > guide on protobuf 3.
> > > > >
> > > > > The fstat error may be related to the build machine.
> > > > >
> > > > > Cheers
> > > > >
> > > > > On Sat, Oct 8, 2016 at 12:18 PM, Ted Yu 
> wrote:
> > > > >
> > > > > > This error didn't appear when the v1 was committed.
> > > > > >
> > > > > > Let me see if I can reproduce it locally on Linux.
> > > > > >
> > > > > > [ERROR] Failed to execute goal org.asciidoctor:asciidoctor-
> > > > > maven-plugin:1.5.3:process-asciidoc (output-pdf) on project hbase:
> > > > > Execution output-pdf of goal org.asciidoctor:asciidoctor-
> > > > > maven-plugin:1.5.3:process-asciidoc failed: org.jruby.exceptions.
> > > > RaiseException:
> > > > > (NotImplementedError) fstat unimplemented unsupported or native
> > support
> > > > > failed to load -> [Help 1]
> > > > > >
> > > > > >
> > > > > > On Sat, Oct 8, 2016 at 11:55 AM, Dima Spivak <
> > dimaspi...@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > >> Looks like HBASE-16750 again. Please revert this again, Ted, and
> > > close
> > > > > as
> > > > > >> "won't fix" if we can'

Re: Still Failing: HBase Generate Website

[张铎]

This maybe related to a jruby issue.

https://github.com/jruby/jruby/issues/2913

It depends on the environment thus it may fail on some machines while
success on some other machines.

2016-10-09 9:29 GMT+08:00 Ted Yu :

> This was the build 16750 went into:
>
> https://builds.apache.org/job/hbase_generate_website/368/
>
> which was green.
>
>
> On Sat, Oct 8, 2016 at 6:10 PM, Sean Busbey  wrote:
>
> > If the website can't build on the build machine then it can't build.
> >
> > We've worked hard as a community to automate this stuff and I don't want
> to
> > go backwards.
> >
> > --
> > Sean Busbey
> >
> > On Oct 8, 2016 14:48, "Ted Yu"  wrote:
> >
> > > I did a local build:
> > >
> > > Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> > > 2015-11-10T16:41:47+00:00)
> > > Maven home: /apache-maven-3.3.9
> > > Java version: 1.8.0_101, vendor: Oracle Corporation
> > > Java home: /jdk1.8.0_101/jre
> > > Default locale: en_US, platform encoding: UTF-8
> > > OS name: "linux", version: "3.10.0-327.28.2.el7.x86_64", arch: "amd64",
> > > family: "unix"
> > >
> > > [INFO] Final Memory: 30M/1928M
> > > [INFO]
> > > 
> 
> > > [INFO] Archetype project created in
> > > /hbase/hbase-archetypes/hbase-client-project/target/build-
> > > archetype/target/generated-sources/archetype
> > > [INFO]
> > > 
> 
> > > [INFO] BUILD SUCCESS
> > > [INFO]
> > > 
> 
> > > [INFO] Total time: 3.168 s
> > >
> > > $ ls -l ./target/site/apache_hbase_reference_guide.pdf
> > > -rw-rw-r-- 1 hbase hbase 13535029 Oct  8 19:35
> > > ./target/site/apache_hbase_reference_guide.pdf
> > >
> > > I can attach the generated pdf to the JIRA if you want. It contains the
> > new
> > > guide on protobuf 3.
> > >
> > > The fstat error may be related to the build machine.
> > >
> > > Cheers
> > >
> > > On Sat, Oct 8, 2016 at 12:18 PM, Ted Yu  wrote:
> > >
> > > > This error didn't appear when the v1 was committed.
> > > >
> > > > Let me see if I can reproduce it locally on Linux.
> > > >
> > > > [ERROR] Failed to execute goal org.asciidoctor:asciidoctor-
> > > maven-plugin:1.5.3:process-asciidoc (output-pdf) on project hbase:
> > > Execution output-pdf of goal org.asciidoctor:asciidoctor-
> > > maven-plugin:1.5.3:process-asciidoc failed: org.jruby.exceptions.
> > RaiseException:
> > > (NotImplementedError) fstat unimplemented unsupported or native support
> > > failed to load -> [Help 1]
> > > >
> > > >
> > > > On Sat, Oct 8, 2016 at 11:55 AM, Dima Spivak 
> > > > wrote:
> > > >
> > > >> Looks like HBASE-16750 again. Please revert this again, Ted, and
> close
> > > as
> > > >> "won't fix" if we can't test it before pushing commits.
> > > >>
> > > >> On Saturday, October 8, 2016, Apache Jenkins Server <
> > > >> jenk...@builds.apache.org> wrote:
> > > >>
> > > >> > Build status: Still Failing
> > > >> >
> > > >> > If successful, the website and docs have been generated. To update
> > the
> > > >> > live site, follow the instructions below. If failed, skip to the
> > > bottom
> > > >> of
> > > >> > this email.
> > > >> >
> > > >> > Use the following commands to download the patch and apply it to a
> > > clean
> > > >> > branch based on origin/asf-site. If you prefer to keep the
> > hbase-site
> > > >> repo
> > > >> > around permanently, you can skip the clone step.
> > > >> >
> > > >> >   git clone https://git-wip-us.apache.org/
> repos/asf/hbase-site.git
> > > >> >
> > > >> >   cd hbase-site
> > > >> >   wget -O- https://builds.apache.org/job/
> > hbase_generate_website/370/
> > > >> > artifact/website.patch.zip | funzip > ${GIT_SHA}.patch
> > > >> >   git fetch
> > > >> >   git checkout -b asf-site-${GIT_SHA} origin/asf-site
> > > >> >   git am --whitespace=fix $GIT_SHA.patch
> > > >> >
> > > >> > At this point, you can preview the changes by opening index.html
> or
> > > any
> > > >> of
> > > >> > the other HTML pages in your local asf-site-${GIT_SHA} branch.
> > > >> >
> > > >> > There are lots of spurious changes, such as timestamps and CSS
> > styles
> > > in
> > > >> > tables, so a generic git diff is not very useful. To see a list of
> > > files
> > > >> > that have been added, deleted, renamed, changed type, or are
> > otherwise
> > > >> > interesting, use the following command:
> > > >> >
> > > >> >   git diff --name-status --diff-filter=ADCRTXUB origin/asf-site
> > > >> >
> > > >> > To see only files that had 100 or more lines changed:
> > > >> >
> > > >> >   git diff --stat origin/asf-site | grep -E '[1-9][0-9]{2,}'
> > > >> >
> > > >> > When you are satisfied, publish your changes to origin/asf-site
> > using
> > > >> > these commands:
> > > >> >
> > > >> >   git commit --allow-empty -m "Empty commit" # to work around a
> > > current
> > > >> > ASF INFRA bug
> > > >> >   git push origin asf-site-${GIT_SHA}:asf-site
> > > >>

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

[张铎]

2016-10-03 1:18 GMT+08:00 Andrew Purtell :

> I have 2.5 and 3.0 installed as /opt/protobuf-, and have bash
> scripts that add the appropriate version's bin directory to the path. Not
> particularly onerous as I also have to switch between required JDK
> versions, so the scripts also set JAVA_HOME at and add JDK bin to the path
> for the required JDK for the build.
>
> Unlike with the scala compiler, which is after all JVM bytecode at a
> fundamental level, I don't think maven automation for automatic download
> and execution is possible. protoc is a native binary.
>
There are precompile binaries for protoc on maven. See here

https://repo.maven.apache.org/maven2/com/google/protobuf/protoc/3.1.0/

I have tested it before. I do not have protoc3 installed but I can compile
this project.

https://github.com/Apache9/grpc-test/blob/master/pom.xml

I think we could optimize the protobuf stuffs later?

>
> > On Oct 1, 2016, at 11:30 PM, 张铎  wrote:
> >
> > Do we need to install protoc 3.0 manully before building HBase or the
> maven
> > protobuf plugin will automatically download the protoc compiler? Maybe we
> > need to install protoc 3.0 in the precommit docker file.
> >
> > 2016-10-02 14:20 GMT+08:00 张铎 :
> >
> >>
> >> 2016-10-02 0:50 GMT+08:00 Stack :
> >>
> >>>> On Sat, Oct 1, 2016 at 7:20 AM, 张铎  wrote:
> >>>>
> >>>> Can we setup a compatibility checker job in jenkins? Start a
> >>> minicluster in
> >>>> one process, and use a client in another process to communicate with
> it.
> >>>> The version of the client should be >= 0.98 and <= the version of the
> >>>> minicluster. Of course we need to design the testing code carefully to
> >>> make
> >>>> sure that we have tested all the cases.
> >>>>
> >>>>
> >>> +1. We need this up and running before we put out an hbase-2.0.0. I
> know
> >>> Matteo does this test manually on a regular basis but a formalization
> >>> would
> >>> help. I can add an exercise of Coprocessor Endpoints. I believe this
> is on
> >>> Dima's list of TODOs but will let him speak for himself.
> >>>
> >>>
> >>>> And also I think we should make sure that no proto3 only feature is
> >>>> introduced in our proto files until branch-1 is dead. Maybe a
> precommit
> >>>> check?
> >>>>
> >>>>
> >>> I think you mean wire-format breaking changes?  Agree. We have our PB3
> set
> >>> to 2.5 compat mode and yes, we can't move on from this until we are in
> a
> >>> place where we can say no to 2.5 clients.
> >>>
> >> Yes, for example, pb2.5 does not support map so we should not use map in
> >> our proto files.
> >>
> >>>
> >>> Making use of PB3isms cannot be avoided. PB3.1 adds a native
> replacement
> >>> for our HBaseZeroCopyByteString/ByteStringer hack. It also adds
> 'unsafe'
> >>> methods that we need to exploit if we are to keep our read/write paths
> >>> offheap.
> >>>
> >>> St.Ack
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>> Thanks.
> >>>>
> >>>> 2016-10-01 11:55 GMT+08:00 Sean Busbey :
> >>>>
> >>>>> have we experimentally confirmed that wire compatibility is
> >>>>> maintained? I saw one mention of expecting wire compatibility to be
> >>>>> fine, but nothing with someone using e.g. the clusterdock work or
> >>>>> something to mix servers / clients or do replication.
> >>>>>
> >>>>>> On Fri, Sep 30, 2016 at 6:30 PM, Stack  wrote:
> >>>>>> I intend to do a mass commit late this weekend that moves us on to a
> >>>>> shaded
> >>>>>> protobuf-3.1.0, either Sunday night or Monday morning.
> >>>>>>
> >>>>>> If objection, please speak up or if need more time for
> >>>>>> consideration/review, just shout.
> >>>>>>
> >>>>>> I want to merge the branch HBASE-16264 into master (it is running
> >>> here
> >>>> up
> >>>>>> on jenkins https://builds.apache.org/view/H-L/view/HBase/job/HBASE-
> >>>>> 16264/).
> >>>>>> The branch at HBASE-16264 has three significant bodies-of-work that
> >>>>>&

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

[张铎]

Got it sir.

And for the new module for CPEP, I think we could put all the classes
annotated as InterfaceAudience.COPROC in it. Ideally, the CPEP should only
depend on interfaces and data structures. Of course, there is a long way to
go.

Thanks.

2016-10-04 11:31 GMT+08:00 Stack :

> On Mon, Oct 3, 2016 at 5:40 PM, 张铎  wrote:
>
> > OK, this means we will still use pb2.5 for hbase-protocol? And
> > hbase-protocol-shaded is maintained manually? I still think the precommit
> > job should have the ability to check the hbase-protocol-shaded...
> >
> >
> Yes. For now. Let me test some more. I think moving all protos to be pb3.1
> will work but will do that after this patch lands (with its pb3.1 for
> internal use and pb2.5 for CPEPs, REST, etc.). All on pb3.1 would make for
> a cleaner story.
>
>
>
> > I skimmed the branch, the proto files are placed both in hbase-protocol
> and
> > hbase-protocol-shaded?
>
>
>
> Yes.  A subset are duplicated. I expended some effort minimizing the set
> but our proto files are way to granular. Files like Client.proto and
> HBase.proto are way too big with too many concerns all collected together
> inside them. It makes it tough drawing the line between internal proto use
> and CPEP protos. In the absence, we have a larger overlap between protos
> used internally and those exposed for CPEP use.
>
> Going forward, I'll add to the doc that many small files is better than a
> few big ones.
>
>
>
> > The one in hbase-protocol is only used to support
> > coprocessor? How can we sync them? Also manually? Or they do not need to
> be
> > synced?
> >
> >
> It'd be tough syncing. They are not exactly the same file given the build
> into different locations. It is not the end-of-the world if they aren't
> sync'd; the CPEP will keep working. Better is that we abandon
> hbase-protocol and start up a new module with published protos that we will
> honor with the annotation public.
>
> This project has revealed a bunch of API surface previously was
> underground.
>
> Let me file some issues.
>
> Thanks Duo,
>
> St.Ack
>
>
>
> > Thanks.
> >
> > 2016-10-04 2:09 GMT+08:00 Stack :
> >
> > > On Mon, Oct 3, 2016 at 9:41 AM, Ted Yu  wrote:
> > >
> > > > The precommit job uses compile-protobuf profile for verification. The
> > > > absence of compile-protobuf profile in hbase-protocol-shaded module
> > means
> > > > precommit job would only invoke the existing compile-protobuf profile
> > > > in hbase-protocol
> > > > module.
> > > >
> > > w.r.t. path, when two protoc executables (corresponding to 2.5 and 3.x,
> > > > respectively) are available, would maven know which one to pick for
> > > > the hbase-protocol and hbase-protocol-shaded modules ?
> > > >
> > > >
> > >
> > > For the former, right, nothing changed.
> > >
> >
> > > On the latter, it is a contrivance, a problem we do not have. The build
> > > doesn't have to pick between pb 2.5 vs pb 3.1. We don't run protoc as
> > part
> > > of our build. It is done apart from the build by the developer and
> their
> > > results are then checked-in. That continues as it was post-patch. We
> just
> > > do 3.1 now instead of 2.5.
> > >
> > > I'll check in a bit of doc as part of this commit that hopefully will
> > help
> > > make this all clearer.
> > >
> > > St.Ack
> > >
> > >
> > >
> > >
> > >
> > > > Cheers
> > > >
> > > > On Mon, Oct 3, 2016 at 9:32 AM, Stack  wrote:
> > > >
> > > > > On Mon, Oct 3, 2016 at 7:16 AM, Ted Yu 
> wrote:
> > > > >
> > > > > > Looks like compile-protobuf profile is not in
> > > > > hbase-protocol-shaded/pom.xml
> > > > > > (in HBASE-16264 branch)
> > > > > >
> > > > > >
> > > > > Sorry. I don't get what you are saying here
> > > > >
> > > > > (The target in the new module is generate-sources. See the included
> > > > README.
> > > > > This step does more work now more than just generating protocs,
> hence
> > > new
> > > > > profile name.)
> > > > >
> > > > >
> > > > >
> > > > > > Seems precommit jobs should pass with the current formation.
> > > > > >
> > > > > >
> &g

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

[张铎]

OK, this means we will still use pb2.5 for hbase-protocol? And
hbase-protocol-shaded is maintained manually? I still think the precommit
job should have the ability to check the hbase-protocol-shaded...

I skimmed the branch, the proto files are placed both in hbase-protocol and
hbase-protocol-shaded? The one in hbase-protocol is only used to support
coprocessor? How can we sync them? Also manually? Or they do not need to be
synced?

Thanks.

2016-10-04 2:09 GMT+08:00 Stack :

> On Mon, Oct 3, 2016 at 9:41 AM, Ted Yu  wrote:
>
> > The precommit job uses compile-protobuf profile for verification. The
> > absence of compile-protobuf profile in hbase-protocol-shaded module means
> > precommit job would only invoke the existing compile-protobuf profile
> > in hbase-protocol
> > module.
> >
> w.r.t. path, when two protoc executables (corresponding to 2.5 and 3.x,
> > respectively) are available, would maven know which one to pick for
> > the hbase-protocol and hbase-protocol-shaded modules ?
> >
> >
>
> For the former, right, nothing changed.
>

> On the latter, it is a contrivance, a problem we do not have. The build
> doesn't have to pick between pb 2.5 vs pb 3.1. We don't run protoc as part
> of our build. It is done apart from the build by the developer and their
> results are then checked-in. That continues as it was post-patch. We just
> do 3.1 now instead of 2.5.
>
> I'll check in a bit of doc as part of this commit that hopefully will help
> make this all clearer.
>
> St.Ack
>
>
>
>
>
> > Cheers
> >
> > On Mon, Oct 3, 2016 at 9:32 AM, Stack  wrote:
> >
> > > On Mon, Oct 3, 2016 at 7:16 AM, Ted Yu  wrote:
> > >
> > > > Looks like compile-protobuf profile is not in
> > > hbase-protocol-shaded/pom.xml
> > > > (in HBASE-16264 branch)
> > > >
> > > >
> > > Sorry. I don't get what you are saying here
> > >
> > > (The target in the new module is generate-sources. See the included
> > README.
> > > This step does more work now more than just generating protocs, hence
> new
> > > profile name.)
> > >
> > >
> > >
> > > > Seems precommit jobs should pass with the current formation.
> > > >
> > > >
> > > Are you stating that this patch is likely to build? (Yes, the patch I
> > > submitted builds).
> > >
> > >
> > >
> > > > In the future, if we add another profile for compiling proto3 files,
> we
> > > > need to specify the path to proto3 compiler.
> > > >
> > > >
> > >
> > > > Please correct me if I am wrong.
> > > >
> > > >
> > > I don't know what you are asking. Why do we have to specify 'paths'? We
> > > don't have to currently (See the plugin we use generating protos now
> from
> > > hadoop). Maybe you are trying to distinguish the production of
> > protobuf2.5
> > > vs 3.1 protos but these are isolated by module
> > >
> > >
> > > I said I'd commit this morning but let me wait a while. There may be
> some
> > > more questions/objections and I'd like to have a clean build up on
> > jenkins
> > > here [1] before I commit (jenkins is being ornery).
> > >
> > > St.Ack
> > > 1.
> > > https://builds.apache.org/job/HBASE-16264/jdk=JDK%201.8%20(
> > > latest),label=yahoo-not-h2/28/console
> > >
> > > Service Unavailable
> > >
> > > The server is temporarily unable to service your
> > > request due to maintenance downtime or capacity
> > > problems. Please try again later.
> > >
> > >
> > >
> > >
> > >
> > >
> > > > On Mon, Oct 3, 2016 at 6:58 AM, Ted Yu  wrote:
> > > >
> > > > > The protoc generated files (such as MasterProtos) are checked into
> > > source
> > > > > repo, right ?
> > > > >
> > > > > Do we need proto3 on the precommit image(s) ?
> > > > >
> > > > > On Mon, Oct 3, 2016 at 5:18 AM, 张铎  wrote:
> > > > >
> > > > >> Then I think we need to file an issue to change the protoc version
> > > > >> installed in the precommit docker file to 3.x before the merge.
> > > > Otherwise
> > > > >> the precommit build for protoc check maybe broken after the
> merge...
> > > > >>
> > > > >>
> > > > &g

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

[张铎]

Then I think we need to file an issue to change the protoc version
installed in the precommit docker file to 3.x before the merge. Otherwise
the precommit build for protoc check maybe broken after the merge...

2016-10-03 1:18 GMT+08:00 Andrew Purtell :

> I have 2.5 and 3.0 installed as /opt/protobuf-, and have bash
> scripts that add the appropriate version's bin directory to the path. Not
> particularly onerous as I also have to switch between required JDK
> versions, so the scripts also set JAVA_HOME at and add JDK bin to the path
> for the required JDK for the build.
>
> Unlike with the scala compiler, which is after all JVM bytecode at a
> fundamental level, I don't think maven automation for automatic download
> and execution is possible. protoc is a native binary.
>
> > On Oct 1, 2016, at 11:30 PM, 张铎  wrote:
> >
> > Do we need to install protoc 3.0 manully before building HBase or the
> maven
> > protobuf plugin will automatically download the protoc compiler? Maybe we
> > need to install protoc 3.0 in the precommit docker file.
> >
> > 2016-10-02 14:20 GMT+08:00 张铎 :
> >
> >>
> >> 2016-10-02 0:50 GMT+08:00 Stack :
> >>
> >>>> On Sat, Oct 1, 2016 at 7:20 AM, 张铎  wrote:
> >>>>
> >>>> Can we setup a compatibility checker job in jenkins? Start a
> >>> minicluster in
> >>>> one process, and use a client in another process to communicate with
> it.
> >>>> The version of the client should be >= 0.98 and <= the version of the
> >>>> minicluster. Of course we need to design the testing code carefully to
> >>> make
> >>>> sure that we have tested all the cases.
> >>>>
> >>>>
> >>> +1. We need this up and running before we put out an hbase-2.0.0. I
> know
> >>> Matteo does this test manually on a regular basis but a formalization
> >>> would
> >>> help. I can add an exercise of Coprocessor Endpoints. I believe this
> is on
> >>> Dima's list of TODOs but will let him speak for himself.
> >>>
> >>>
> >>>> And also I think we should make sure that no proto3 only feature is
> >>>> introduced in our proto files until branch-1 is dead. Maybe a
> precommit
> >>>> check?
> >>>>
> >>>>
> >>> I think you mean wire-format breaking changes?  Agree. We have our PB3
> set
> >>> to 2.5 compat mode and yes, we can't move on from this until we are in
> a
> >>> place where we can say no to 2.5 clients.
> >>>
> >> Yes, for example, pb2.5 does not support map so we should not use map in
> >> our proto files.
> >>
> >>>
> >>> Making use of PB3isms cannot be avoided. PB3.1 adds a native
> replacement
> >>> for our HBaseZeroCopyByteString/ByteStringer hack. It also adds
> 'unsafe'
> >>> methods that we need to exploit if we are to keep our read/write paths
> >>> offheap.
> >>>
> >>> St.Ack
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>> Thanks.
> >>>>
> >>>> 2016-10-01 11:55 GMT+08:00 Sean Busbey :
> >>>>
> >>>>> have we experimentally confirmed that wire compatibility is
> >>>>> maintained? I saw one mention of expecting wire compatibility to be
> >>>>> fine, but nothing with someone using e.g. the clusterdock work or
> >>>>> something to mix servers / clients or do replication.
> >>>>>
> >>>>>> On Fri, Sep 30, 2016 at 6:30 PM, Stack  wrote:
> >>>>>> I intend to do a mass commit late this weekend that moves us on to a
> >>>>> shaded
> >>>>>> protobuf-3.1.0, either Sunday night or Monday morning.
> >>>>>>
> >>>>>> If objection, please speak up or if need more time for
> >>>>>> consideration/review, just shout.
> >>>>>>
> >>>>>> I want to merge the branch HBASE-16264 into master (it is running
> >>> here
> >>>> up
> >>>>>> on jenkins https://builds.apache.org/view/H-L/view/HBase/job/HBASE-
> >>>>> 16264/).
> >>>>>> The branch at HBASE-16264 has three significant bodies-of-work that
> >>>>>> unfortunately are tangled and can only go in of a piece.
> >>>>>>
> >>>>>> *

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

[张铎]

Do we need to install protoc 3.0 manully before building HBase or the maven
protobuf plugin will automatically download the protoc compiler? Maybe we
need to install protoc 3.0 in the precommit docker file.

2016-10-02 14:20 GMT+08:00 张铎 :

>
> 2016-10-02 0:50 GMT+08:00 Stack :
>
>> On Sat, Oct 1, 2016 at 7:20 AM, 张铎  wrote:
>>
>> > Can we setup a compatibility checker job in jenkins? Start a
>> minicluster in
>> > one process, and use a client in another process to communicate with it.
>> > The version of the client should be >= 0.98 and <= the version of the
>> > minicluster. Of course we need to design the testing code carefully to
>> make
>> > sure that we have tested all the cases.
>> >
>> >
>> +1. We need this up and running before we put out an hbase-2.0.0. I know
>> Matteo does this test manually on a regular basis but a formalization
>> would
>> help. I can add an exercise of Coprocessor Endpoints. I believe this is on
>> Dima's list of TODOs but will let him speak for himself.
>>
>>
>> > And also I think we should make sure that no proto3 only feature is
>> > introduced in our proto files until branch-1 is dead. Maybe a precommit
>> > check?
>> >
>> >
>> I think you mean wire-format breaking changes?  Agree. We have our PB3 set
>> to 2.5 compat mode and yes, we can't move on from this until we are in a
>> place where we can say no to 2.5 clients.
>>
> Yes, for example, pb2.5 does not support map so we should not use map in
> our proto files.
>
>>
>> Making use of PB3isms cannot be avoided. PB3.1 adds a native replacement
>> for our HBaseZeroCopyByteString/ByteStringer hack. It also adds 'unsafe'
>> methods that we need to exploit if we are to keep our read/write paths
>> offheap.
>>
>> St.Ack
>>
>>
>>
>>
>>
>> > Thanks.
>> >
>> > 2016-10-01 11:55 GMT+08:00 Sean Busbey :
>> >
>> > > have we experimentally confirmed that wire compatibility is
>> > > maintained? I saw one mention of expecting wire compatibility to be
>> > > fine, but nothing with someone using e.g. the clusterdock work or
>> > > something to mix servers / clients or do replication.
>> > >
>> > > On Fri, Sep 30, 2016 at 6:30 PM, Stack  wrote:
>> > > > I intend to do a mass commit late this weekend that moves us on to a
>> > > shaded
>> > > > protobuf-3.1.0, either Sunday night or Monday morning.
>> > > >
>> > > > If objection, please speak up or if need more time for
>> > > > consideration/review, just shout.
>> > > >
>> > > > I want to merge the branch HBASE-16264 into master (it is running
>> here
>> > up
>> > > > on jenkins https://builds.apache.org/view/H-L/view/HBase/job/HBASE-
>> > > 16264/).
>> > > > The branch at HBASE-16264 has three significant bodies-of-work that
>> > > > unfortunately are tangled and can only go in of a piece.
>> > > >
>> > > >  * HBASE-16264 <https://issues.apache.org/jira/browse/HBASE-16264>
>> The
>> > > > shading of our protobuf usage so we can upgrade and/or run with a
>> > patched
>> > > > protobuf WITHOUT breaking REST, Spark, and in particular,
>> Coprocessor
>> > > > Endpoints.
>> > > >  * HBASE-16567 <https://issues.apache.org/jira/browse/HBASE-16567>
>> A
>> > > move
>> > > > up on to (shaded) protobuf-3.1.0
>> > > >  * HBASE-16741 <https://issues.apache.org/jira/browse/HBASE-16741>
>> An
>> > > > amendment of our generate protobufs step to include shading and a
>> > > bundling
>> > > > of protobuf src (with a means of calling a patch srcs hook)
>> > > >
>> > > > Together we're talking about 40MB of change mostly made of the
>> movement
>> > > of
>> > > > generated files or the application of a pattern that alters where we
>> > get
>> > > > imports from. When done, you should notice no difference and should
>> be
>> > > able
>> > > > to go about your business as per usual. Upside is that we will be
>> able
>> > to
>> > > > avoid coming onheap doing protobuf marshalling/unmarshalling as
>> > protobuf
>> > > > 2.5.0 requires. Downside is that we repeat a good portion of our
>> > interna

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

[张铎]

2016-10-02 0:50 GMT+08:00 Stack :

> On Sat, Oct 1, 2016 at 7:20 AM, 张铎  wrote:
>
> > Can we setup a compatibility checker job in jenkins? Start a minicluster
> in
> > one process, and use a client in another process to communicate with it.
> > The version of the client should be >= 0.98 and <= the version of the
> > minicluster. Of course we need to design the testing code carefully to
> make
> > sure that we have tested all the cases.
> >
> >
> +1. We need this up and running before we put out an hbase-2.0.0. I know
> Matteo does this test manually on a regular basis but a formalization would
> help. I can add an exercise of Coprocessor Endpoints. I believe this is on
> Dima's list of TODOs but will let him speak for himself.
>
>
> > And also I think we should make sure that no proto3 only feature is
> > introduced in our proto files until branch-1 is dead. Maybe a precommit
> > check?
> >
> >
> I think you mean wire-format breaking changes?  Agree. We have our PB3 set
> to 2.5 compat mode and yes, we can't move on from this until we are in a
> place where we can say no to 2.5 clients.
>
Yes, for example, pb2.5 does not support map so we should not use map in
our proto files.

>
> Making use of PB3isms cannot be avoided. PB3.1 adds a native replacement
> for our HBaseZeroCopyByteString/ByteStringer hack. It also adds 'unsafe'
> methods that we need to exploit if we are to keep our read/write paths
> offheap.
>
> St.Ack
>
>
>
>
>
> > Thanks.
> >
> > 2016-10-01 11:55 GMT+08:00 Sean Busbey :
> >
> > > have we experimentally confirmed that wire compatibility is
> > > maintained? I saw one mention of expecting wire compatibility to be
> > > fine, but nothing with someone using e.g. the clusterdock work or
> > > something to mix servers / clients or do replication.
> > >
> > > On Fri, Sep 30, 2016 at 6:30 PM, Stack  wrote:
> > > > I intend to do a mass commit late this weekend that moves us on to a
> > > shaded
> > > > protobuf-3.1.0, either Sunday night or Monday morning.
> > > >
> > > > If objection, please speak up or if need more time for
> > > > consideration/review, just shout.
> > > >
> > > > I want to merge the branch HBASE-16264 into master (it is running
> here
> > up
> > > > on jenkins https://builds.apache.org/view/H-L/view/HBase/job/HBASE-
> > > 16264/).
> > > > The branch at HBASE-16264 has three significant bodies-of-work that
> > > > unfortunately are tangled and can only go in of a piece.
> > > >
> > > >  * HBASE-16264 <https://issues.apache.org/jira/browse/HBASE-16264>
> The
> > > > shading of our protobuf usage so we can upgrade and/or run with a
> > patched
> > > > protobuf WITHOUT breaking REST, Spark, and in particular, Coprocessor
> > > > Endpoints.
> > > >  * HBASE-16567 <https://issues.apache.org/jira/browse/HBASE-16567> A
> > > move
> > > > up on to (shaded) protobuf-3.1.0
> > > >  * HBASE-16741 <https://issues.apache.org/jira/browse/HBASE-16741>
> An
> > > > amendment of our generate protobufs step to include shading and a
> > > bundling
> > > > of protobuf src (with a means of calling a patch srcs hook)
> > > >
> > > > Together we're talking about 40MB of change mostly made of the
> movement
> > > of
> > > > generated files or the application of a pattern that alters where we
> > get
> > > > imports from. When done, you should notice no difference and should
> be
> > > able
> > > > to go about your business as per usual. Upside is that we will be
> able
> > to
> > > > avoid coming onheap doing protobuf marshalling/unmarshalling as
> > protobuf
> > > > 2.5.0 requires. Downside is that we repeat a good portion of our
> > internal
> > > > protos, once non-shaded so Coprocessor Endpoints can keep working and
> > > then
> > > > again as shaded for internal use.
> > > >
> > > > I provide some more overview below on the changes. See the shading
> doc
> > > > here:
> > > > https://docs.google.com/document/d/1H4NgLXQ9Y9KejwobddCqaVMEDCGby
> > > DcXtdF5iAfDIEk/edit#
> > > > for more detail (Patches are up on review board -- except the latest
> > > > HBASE-16264 which is too big for JIRA and RB). I am currently working
> > on
> > > a
> > > &

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

[张铎]

Can we setup a compatibility checker job in jenkins? Start a minicluster in
one process, and use a client in another process to communicate with it.
The version of the client should be >= 0.98 and <= the version of the
minicluster. Of course we need to design the testing code carefully to make
sure that we have tested all the cases.

And also I think we should make sure that no proto3 only feature is
introduced in our proto files until branch-1 is dead. Maybe a precommit
check?

Thanks.

2016-10-01 11:55 GMT+08:00 Sean Busbey :

> have we experimentally confirmed that wire compatibility is
> maintained? I saw one mention of expecting wire compatibility to be
> fine, but nothing with someone using e.g. the clusterdock work or
> something to mix servers / clients or do replication.
>
> On Fri, Sep 30, 2016 at 6:30 PM, Stack  wrote:
> > I intend to do a mass commit late this weekend that moves us on to a
> shaded
> > protobuf-3.1.0, either Sunday night or Monday morning.
> >
> > If objection, please speak up or if need more time for
> > consideration/review, just shout.
> >
> > I want to merge the branch HBASE-16264 into master (it is running here up
> > on jenkins https://builds.apache.org/view/H-L/view/HBase/job/HBASE-
> 16264/).
> > The branch at HBASE-16264 has three significant bodies-of-work that
> > unfortunately are tangled and can only go in of a piece.
> >
> >  * HBASE-16264  The
> > shading of our protobuf usage so we can upgrade and/or run with a patched
> > protobuf WITHOUT breaking REST, Spark, and in particular, Coprocessor
> > Endpoints.
> >  * HBASE-16567  A
> move
> > up on to (shaded) protobuf-3.1.0
> >  * HBASE-16741  An
> > amendment of our generate protobufs step to include shading and a
> bundling
> > of protobuf src (with a means of calling a patch srcs hook)
> >
> > Together we're talking about 40MB of change mostly made of the movement
> of
> > generated files or the application of a pattern that alters where we get
> > imports from. When done, you should notice no difference and should be
> able
> > to go about your business as per usual. Upside is that we will be able to
> > avoid coming onheap doing protobuf marshalling/unmarshalling as protobuf
> > 2.5.0 requires. Downside is that we repeat a good portion of our internal
> > protos, once non-shaded so Coprocessor Endpoints can keep working and
> then
> > again as shaded for internal use.
> >
> > I provide some more overview below on the changes. See the shading doc
> > here:
> > https://docs.google.com/document/d/1H4NgLXQ9Y9KejwobddCqaVMEDCGby
> DcXtdF5iAfDIEk/edit#
> > for more detail (Patches are up on review board -- except the latest
> > HBASE-16264 which is too big for JIRA and RB). I am currently working on
> a
> > devs chapter for the book on protobuf going forward that will go in as
> part
> > of this patch.
> >
> > Thanks,
> > St.Ack
> >
> > Items of note:
> >
> >  * Two new modules; one named hbase-protocol-shaded that is used by hbase
> > core. It has in it a shaded (and later patched) protobuf. The other new
> > module is hbase-endpoint which goes after hbase-server and has those
> > bundled endpoints that I was able to break out of core (there are a few
> > that are hopelessly entangled that need to be undone as CPEPs but
> > fortunately belong in core: Auth, Access, MultiRow).
> >  * I've tested running a branch-1 CPEP against a master with these
> patches
> > in place and stuff like ACL (A CPEP) run from the branch-1 shell work
> > against the branch-2 server.
> >
> >
> >
> >
> >
> > On Mon, Aug 22, 2016 at 5:20 PM, Stack  wrote:
> >
> >> This project goes on. I updated HBASE-1563 "Shade protobuf" with some
> doc
> >> on a final approach. We need to be able to refer to both shaded and
> >> non-shaded protobuf so we can support sending HDFS old-school pb
> Messages
> >> but also so Coprocessor Endpoints keep working though internally
> protobufs
> >> have been relocated. Funny you should ask, but yes, there are some
> >> downsides (as predicted by contributors on the JIRA). I'd be interested
> to
> >> hear if they are too burdensome. In particular, your IDE experience
> gets a
> >> little convoluted as you will need to add to your build path, a jar with
> >> the relocated pbs. A pain.
> >>
> >> Thanks,
> >> St.Ack
> >>
> >>
> >> On Wed, Apr 13, 2016 at 6:09 AM, Stack  wrote:
> >>
> >>> On Tue, Apr 12, 2016 at 9:26 PM, Sean Busbey 
> wrote:
> >>>
>  On Tue, Apr 12, 2016 at 6:17 PM, Stack  wrote:
>  >
>  >
>  > On an initial pass, the only difficult part seems to be interaction
>  with
>  > HDFS in asyncwal (might just pull in the HDFS messages).
>  >
>  >
> 
>  I have some idea how we can make this work either by pushing asyncwal
>  upstream to HDFS or through some maven tricks, depending on how much
>  time we have.
> 

Re: [DISCUSSION] MR jobs started by Master or RS

[张铎]

No, not your fault, at lease, not this time:)

Why I call the code ugly? Can you simply tell me the sequence of calls when
starting up the HMaster? HMaster is also a regionserver so it extends
HRegionServer, and the initialization of HRegionServer sometimes needs to
make rpc calls to HMaster. A simple change would cause probabilistic dead
lock or some strange NPEs...

That's why I'm very nervous when somebody wants to add new features or add
external dependencies to HMaster, especially add more works for the start
up processing...

Thanks.

2016-09-23 20:02 GMT+08:00 Ted Yu :

> I read through HADOOP-13433
> <https://issues.apache.org/jira/browse/HADOOP-13433> - the cited race
> condition is in jdk.
>
> Suggest pinging the reviewer on JIRA to get it moving.
>
> bq. But the ugly code in HMaster is readlly a problem...
>
> Can you be specific as to which code is ugly ? Is it in the backup /
> restore mega patch ?
>
> Cheers
>
> On Thu, Sep 22, 2016 at 10:44 PM, 张铎  wrote:
>
> > If you guys have already implemented the feature in the MR way and the
> > patch is ready for landing on master, I'm a -0 on it as I do not want to
> > block the development progress.
> >
> > But I strongly suggest later we need to revisit the design and see if we
> > can seperated the logic from HMaster as much as possible. HA is not a big
> > problem if you do not store any metada locally. But the ugly code in
> > HMaster is readlly a problem...
> >
> > And for security, I have a issue pending for a long time. Can someone
> help
> > taking a simple look at it? This is what I mean, ugly code... logout and
> > destroy the credentials in a subject when it is still being used, and
> > declared as LimitPrivacy so I can not change the behivor and the only way
> > to fix it is to write another piece of ugly code...
> >
> > https://issues.apache.org/jira/browse/HADOOP-13433
> >
> > 2016-09-23 12:53 GMT+08:00 Vladimir Rodionov :
> >
> > > >> If in the future, we find better ways of doing this without using
> MR,
> > we
> > > can certainly consider that
> > >
> > > Our framework for distributed operations is abstract and allows
> > > different implementations. MR is just one implementation we provide.
> > >
> > > -Vlad
> > >
> > > On Thu, Sep 22, 2016 at 9:38 PM, Devaraj Das 
> > wrote:
> > >
> > > > Guys, first off apologies for bringing in the topic of MR-based
> > > > compactions.. But I was thinking more about the SpliceMachine
> approach
> > of
> > > > managing compactions in Spark where apparently they saw a lot of
> > > benefits.
> > > > Apologies for giving you that sore throat Andrew; I really didn't
> mean
> > to
> > > > :-)
> > > >
> > > > So on this issue, we have these on the plate:
> > > > 0. Somehow not use MR but something like that
> > > > 1. Run a standalone service other than master
> > > > 2. Shell out from the master
> > > >
> > > > I don't think we have a good answer to (0), and I don't think it's
> even
> > > > worth the effort of trying to build something when MR is already
> there,
> > > and
> > > > being used by HBase already for some operations.
> > > >
> > > > On (1), we have to deal with a myriad of issues - HA of the server
> not
> > > > being the least of them all. Security (kerberos authentication,
> another
> > > > keytab to manage, etc. etc. etc.). IMO, that approach is DOA. Instead
> > > let's
> > > > substitute that (1) with the HBase Master. I haven't seen any good
> > reason
> > > > why the HBase master shouldn't launch MR jobs if needed. It's not
> > ideal;
> > > > agreed.
> > > >
> > > > Now before going to (2), let's see what are the benefits of running
> the
> > > > backup/restore jobs from the master. I think Ted has summarized some
> of
> > > the
> > > > issues that we need to take care of - basically, the master can keep
> > > track
> > > > of running jobs, and should it fail, the backup master can continue
> > > keeping
> > > > track of it (since the jobId would have been recorded in the proc
> WAL).
> > > The
> > > > master can also do cleanup, etc. of failed backup/restore processes.
> > > > Security is another issue - the job needs to run as 'hbase' since it
> > owns
> > > > the

Re: [DISCUSSION] MR jobs started by Master or RS

[张铎]

If you guys have already implemented the feature in the MR way and the
patch is ready for landing on master, I'm a -0 on it as I do not want to
block the development progress.

But I strongly suggest later we need to revisit the design and see if we
can seperated the logic from HMaster as much as possible. HA is not a big
problem if you do not store any metada locally. But the ugly code in
HMaster is readlly a problem...

And for security, I have a issue pending for a long time. Can someone help
taking a simple look at it? This is what I mean, ugly code... logout and
destroy the credentials in a subject when it is still being used, and
declared as LimitPrivacy so I can not change the behivor and the only way
to fix it is to write another piece of ugly code...

https://issues.apache.org/jira/browse/HADOOP-13433

2016-09-23 12:53 GMT+08:00 Vladimir Rodionov :

> >> If in the future, we find better ways of doing this without using MR, we
> can certainly consider that
>
> Our framework for distributed operations is abstract and allows
> different implementations. MR is just one implementation we provide.
>
> -Vlad
>
> On Thu, Sep 22, 2016 at 9:38 PM, Devaraj Das  wrote:
>
> > Guys, first off apologies for bringing in the topic of MR-based
> > compactions.. But I was thinking more about the SpliceMachine approach of
> > managing compactions in Spark where apparently they saw a lot of
> benefits.
> > Apologies for giving you that sore throat Andrew; I really didn't mean to
> > :-)
> >
> > So on this issue, we have these on the plate:
> > 0. Somehow not use MR but something like that
> > 1. Run a standalone service other than master
> > 2. Shell out from the master
> >
> > I don't think we have a good answer to (0), and I don't think it's even
> > worth the effort of trying to build something when MR is already there,
> and
> > being used by HBase already for some operations.
> >
> > On (1), we have to deal with a myriad of issues - HA of the server not
> > being the least of them all. Security (kerberos authentication, another
> > keytab to manage, etc. etc. etc.). IMO, that approach is DOA. Instead
> let's
> > substitute that (1) with the HBase Master. I haven't seen any good reason
> > why the HBase master shouldn't launch MR jobs if needed. It's not ideal;
> > agreed.
> >
> > Now before going to (2), let's see what are the benefits of running the
> > backup/restore jobs from the master. I think Ted has summarized some of
> the
> > issues that we need to take care of - basically, the master can keep
> track
> > of running jobs, and should it fail, the backup master can continue
> keeping
> > track of it (since the jobId would have been recorded in the proc WAL).
> The
> > master can also do cleanup, etc. of failed backup/restore processes.
> > Security is another issue - the job needs to run as 'hbase' since it owns
> > the data. Having the master launch the job makes it get that privilege.
> In
> > the (2) approach, it's hard to do some of the above management.
> >
> > Guys, just to reiterate, the patch as such is ready from the overall
> > design/arch point of view (maybe code review is still pending from
> Matteo).
> > If in the future, we find better ways of doing this without using MR, we
> > can certainly consider that. But IMO don't think we should block this
> patch
> > from getting merged.
> >
> > 
> > From: 张铎 
> > Sent: Thursday, September 22, 2016 8:32 PM
> > To: dev@hbase.apache.org
> > Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> >
> > So what about a standalone service other than master? You can use your
> own
> > procedure store in that service?
> >
> > 2016-09-23 11:28 GMT+08:00 Ted Yu :
> >
> > > An earlier implementation was client driven.
> > >
> > > But with that approach, it is hard to resume if there is error midway.
> > > Using Procedure V2 makes the backup / restore more robust.
> > >
> > > Another consideration is for security. It is hard to enforce security
> (to
> > > be implemented) for client driven actions.
> > >
> > > Cheers
> > >
> > > > On Sep 22, 2016, at 8:15 PM, Andrew Purtell <
> andrew.purt...@gmail.com>
> > > wrote:
> > > >
> > > > No, this misses Matteo's finer point, which is "shelling out" from
> the
> > > master directly to run MR is a first. Why not drive this with a utility
> > > derived from Tool?
>

Re: [DISCUSSION] MR jobs started by Master or RS

[张铎]

2016-09-23 12:38 GMT+08:00 Devaraj Das :

> Guys, first off apologies for bringing in the topic of MR-based
> compactions.. But I was thinking more about the SpliceMachine approach of
> managing compactions in Spark where apparently they saw a lot of benefits.
> Apologies for giving you that sore throat Andrew; I really didn't mean to
> :-)
>
> So on this issue, we have these on the plate:
> 0. Somehow not use MR but something like that
> 1. Run a standalone service other than master
> 2. Shell out from the master
>
> I don't think we have a good answer to (0), and I don't think it's even
> worth the effort of trying to build something when MR is already there, and
> being used by HBase already for some operations.
>
> On (1), we have to deal with a myriad of issues - HA of the server not
> being the least of them all. Security (kerberos authentication, another
> keytab to manage, etc. etc. etc.). IMO, that approach is DOA. Instead let's
> substitute that (1) with the HBase Master. I haven't seen any good reason
> why the HBase master shouldn't launch MR jobs if needed. It's not ideal;
> agreed.
>
> We have already put lots of stuffs in HMaster, especially on startup, we
even need to run the initializations in a background thread to let the
master get up quickly and causes lots of races. I do not think it is a good
idea to keep messing up the code there.
For MR, as I said above, configuration. I do not want to restart HMaster if
I just want to tune the config of a backup MR job. Yes, you could introduce
new shell commands to do that, change job config, change YARN cluster, and
persist the change to some places, maybe zookeeper? Oh no...


> Now before going to (2), let's see what are the benefits of running the
> backup/restore jobs from the master. I think Ted has summarized some of the
> issues that we need to take care of - basically, the master can keep track
> of running jobs, and should it fail, the backup master can continue keeping
> track of it (since the jobId would have been recorded in the proc WAL). The
> master can also do cleanup, etc. of failed backup/restore processes.
> Security is another issue - the job needs to run as 'hbase' since it owns
> the data. Having the master launch the job makes it get that privilege. In
> the (2) approach, it's hard to do some of the above management.
>
> Guys, just to reiterate, the patch as such is ready from the overall
> design/arch point of view (maybe code review is still pending from Matteo).
> If in the future, we find better ways of doing this without using MR, we
> can certainly consider that. But IMO don't think we should block this patch
> from getting merged.
>
> 
> From: 张铎 
> Sent: Thursday, September 22, 2016 8:32 PM
> To: dev@hbase.apache.org
> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
>
> So what about a standalone service other than master? You can use your own
> procedure store in that service?
>
> 2016-09-23 11:28 GMT+08:00 Ted Yu :
>
> > An earlier implementation was client driven.
> >
> > But with that approach, it is hard to resume if there is error midway.
> > Using Procedure V2 makes the backup / restore more robust.
> >
> > Another consideration is for security. It is hard to enforce security (to
> > be implemented) for client driven actions.
> >
> > Cheers
> >
> > > On Sep 22, 2016, at 8:15 PM, Andrew Purtell 
> > wrote:
> > >
> > > No, this misses Matteo's finer point, which is "shelling out" from the
> > master directly to run MR is a first. Why not drive this with a utility
> > derived from Tool?
> > >
> > > On Sep 22, 2016, at 7:57 PM, Vladimir Rodionov  >
> > wrote:
> > >
> > >>>> In our production cluster,  it is a common case we just have HDFS
> and
> > >>>> HBase deployed.
> > >>>> If our Master/RS depend on MR framework (especially some features we
> > >>>> have not used at all),  it introduced another cost for maintain.  I
> > >>>> don't think it is a good idea.
> > >>
> > >> So , you are not backup users in this case. Many our customers have
> full
> > >> stack deployed and
> > >> want see backup to be a standard feature. Besides this, nothing will
> > happen
> > >> in your cluster
> > >> if you won't be doing backups.
> > >>
> > >> This discussion (we do not want see M/R dependency) goes to nowhere.
> We
> > >> asked already, at least twice, to suggest another framework (other
&

Re: [DISCUSSION] MR jobs started by Master or RS

[张铎]

Stability is one thing, and another thing is the difficulty of
configuration and deployment.

For configuration, it is always a pain. I do not want to restart HMaster
many times to get thing right. A standalone service would be better.

For deployment, as chenheng said above, usually we do not deploy a YARN
along with the HBase cluster, which means we need to use an external YARN
cluster to run the job. The cluster may not be controlled by us which means
it may miss the HBase jars or the version is incorrect(as seems YARN's
timeline server depends on HBase?). This could cause the job to fail
because ClassNotFound or some other strange exceptions.

Thanks.

2016-09-23 11:47 GMT+08:00 Ted Yu :

> You mean standalone service which runs Procedure V2 ?
> Not sure how much work is involved.
>
> Is this concerning the stability of Master where backup / restore
> procedures run ?
>
> To my understanding, errors in one procedure are isolated, not having
> adverse impact on Master's stability.
>
> On Thu, Sep 22, 2016 at 8:32 PM, 张铎  wrote:
>
> > So what about a standalone service other than master? You can use your
> own
> > procedure store in that service?
> >
> > 2016-09-23 11:28 GMT+08:00 Ted Yu :
> >
> > > An earlier implementation was client driven.
> > >
> > > But with that approach, it is hard to resume if there is error midway.
> > > Using Procedure V2 makes the backup / restore more robust.
> > >
> > > Another consideration is for security. It is hard to enforce security
> (to
> > > be implemented) for client driven actions.
> > >
> > > Cheers
> > >
> > > > On Sep 22, 2016, at 8:15 PM, Andrew Purtell <
> andrew.purt...@gmail.com>
> > > wrote:
> > > >
> > > > No, this misses Matteo's finer point, which is "shelling out" from
> the
> > > master directly to run MR is a first. Why not drive this with a utility
> > > derived from Tool?
> > > >
> > > > On Sep 22, 2016, at 7:57 PM, Vladimir Rodionov <
> vladrodio...@gmail.com
> > >
> > > wrote:
> > > >
> > > >>>> In our production cluster,  it is a common case we just have HDFS
> > and
> > > >>>> HBase deployed.
> > > >>>> If our Master/RS depend on MR framework (especially some features
> we
> > > >>>> have not used at all),  it introduced another cost for maintain.
> I
> > > >>>> don't think it is a good idea.
> > > >>
> > > >> So , you are not backup users in this case. Many our customers have
> > full
> > > >> stack deployed and
> > > >> want see backup to be a standard feature. Besides this, nothing will
> > > happen
> > > >> in your cluster
> > > >> if you won't be doing backups.
> > > >>
> > > >> This discussion (we do not want see M/R dependency) goes to nowhere.
> > We
> > > >> asked already, at least twice, to suggest another framework (other
> > than
> > > M/R)
> > > >> for bulk data copy with *conversion*. Still waiting for suggestions.
> > > >>
> > > >> -Vlad
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>> On Thu, Sep 22, 2016 at 7:49 PM, Ted Yu 
> wrote:
> > > >>>
> > > >>> If MR framework is not deployed in the cluster, hbase still
> functions
> > > >>> normally (post merge).
> > > >>>
> > > >>> In terms of build time dependency, we have long been depending on
> > > >>> mapreduce. Take a look at ExportSnapshot.
> > > >>>
> > > >>> Cheers
> > > >>>
> > > >>> On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <
> heng.chen.1...@gmail.com
> > >
> > > >>> wrote:
> > > >>>
> > > >>>> In our production cluster,  it is a common case we just have HDFS
> > and
> > > >>>> HBase deployed.
> > > >>>> If our Master/RS depend on MR framework (especially some features
> we
> > > >>>> have not used at all),  it introduced another cost for maintain.
> I
> > > >>>> don't think it is a good idea.
> > > >>>>
> > > >>>> 2016-09-23 10:28 GMT+08:00 张铎 :
> > > >>>>> To be specific, for example, our nice Backup/Restore feature, 

Re: [DISCUSSION] MR jobs started by Master or RS

[张铎]

So what about a standalone service other than master? You can use your own
procedure store in that service?

2016-09-23 11:28 GMT+08:00 Ted Yu :

> An earlier implementation was client driven.
>
> But with that approach, it is hard to resume if there is error midway.
> Using Procedure V2 makes the backup / restore more robust.
>
> Another consideration is for security. It is hard to enforce security (to
> be implemented) for client driven actions.
>
> Cheers
>
> > On Sep 22, 2016, at 8:15 PM, Andrew Purtell 
> wrote:
> >
> > No, this misses Matteo's finer point, which is "shelling out" from the
> master directly to run MR is a first. Why not drive this with a utility
> derived from Tool?
> >
> > On Sep 22, 2016, at 7:57 PM, Vladimir Rodionov 
> wrote:
> >
> >>>> In our production cluster,  it is a common case we just have HDFS and
> >>>> HBase deployed.
> >>>> If our Master/RS depend on MR framework (especially some features we
> >>>> have not used at all),  it introduced another cost for maintain.  I
> >>>> don't think it is a good idea.
> >>
> >> So , you are not backup users in this case. Many our customers have full
> >> stack deployed and
> >> want see backup to be a standard feature. Besides this, nothing will
> happen
> >> in your cluster
> >> if you won't be doing backups.
> >>
> >> This discussion (we do not want see M/R dependency) goes to nowhere. We
> >> asked already, at least twice, to suggest another framework (other than
> M/R)
> >> for bulk data copy with *conversion*. Still waiting for suggestions.
> >>
> >> -Vlad
> >>
> >>
> >>
> >>
> >>> On Thu, Sep 22, 2016 at 7:49 PM, Ted Yu  wrote:
> >>>
> >>> If MR framework is not deployed in the cluster, hbase still functions
> >>> normally (post merge).
> >>>
> >>> In terms of build time dependency, we have long been depending on
> >>> mapreduce. Take a look at ExportSnapshot.
> >>>
> >>> Cheers
> >>>
> >>> On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen 
> >>> wrote:
> >>>
> >>>> In our production cluster,  it is a common case we just have HDFS and
> >>>> HBase deployed.
> >>>> If our Master/RS depend on MR framework (especially some features we
> >>>> have not used at all),  it introduced another cost for maintain.  I
> >>>> don't think it is a good idea.
> >>>>
> >>>> 2016-09-23 10:28 GMT+08:00 张铎 :
> >>>>> To be specific, for example, our nice Backup/Restore feature, if we
> >>> think
> >>>>> this is not a core feature of HBase, then we could make it depend on
> >>> MR,
> >>>>> and start a standalone BackupManager instance that submits MR jobs to
> >>> do
> >>>>> periodical maintenance job. And if we think this is a core feature
> that
> >>>>> everyone should use it, then we'd better implement it without MR
> >>>>> dependency, like DLS.
> >>>>>
> >>>>> Thanks.
> >>>>>
> >>>>> 2016-09-23 10:11 GMT+08:00 张铎 :
> >>>>>
> >>>>>> I‘m -1 on let master or rs launch MR jobs. It is OK that some of our
> >>>>>> features depend on MR but I think the bottom line is that we should
> >>>> launch
> >>>>>> the jobs from outside manually or by other services.
> >>>>>>
> >>>>>> 2016-09-23 9:47 GMT+08:00 Andrew Purtell  >:
> >>>>>>
> >>>>>>> Ok, got it. Well "shelling out" is on the line I think, so a fair
> >>>>>>> question.
> >>>>>>>
> >>>>>>> Can this be driven by a utility derived from Tool like our other MR
> >>>> apps?
> >>>>>>> The issue is needing the AccessController to decide if allowed? But
> >>>> nothing
> >>>>>>> prevents the user from running the job manually/independently,
> right?
> >>>>>>>
> >>>>>>>> On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
> >>>> theo.berto...@gmail.com>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> just a remark. my query was not about to

Re: [DISCUSSION] MR jobs started by Master or RS

[张铎]

To be specific, for example, our nice Backup/Restore feature, if we think
this is not a core feature of HBase, then we could make it depend on MR,
and start a standalone BackupManager instance that submits MR jobs to do
periodical maintenance job. And if we think this is a core feature that
everyone should use it, then we'd better implement it without MR
dependency, like DLS.

Thanks.

2016-09-23 10:11 GMT+08:00 张铎 :

> I‘m -1 on let master or rs launch MR jobs. It is OK that some of our
> features depend on MR but I think the bottom line is that we should launch
> the jobs from outside manually or by other services.
>
> 2016-09-23 9:47 GMT+08:00 Andrew Purtell :
>
>> Ok, got it. Well "shelling out" is on the line I think, so a fair
>> question.
>>
>> Can this be driven by a utility derived from Tool like our other MR apps?
>> The issue is needing the AccessController to decide if allowed? But nothing
>> prevents the user from running the job manually/independently, right?
>>
>> > On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi 
>> wrote:
>> >
>> > just a remark. my query was not about tools using MR (everyone i think
>> is
>> > ok with those).
>> > the topic was about: "are we ok with running MR jobs from Master and RSs
>> > code?" since this will be the first time we do this
>> >
>> > Matteo
>> >
>> >
>> >> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das 
>> wrote:
>> >>
>> >> Very much agree; for tools like ExportSnapshot / Backup / Restore, it's
>> >> fine to be dependent on MR. MR is the right framework for such. We
>> should
>> >> also do compactions using MR (just saying :) )
>> >> 
>> >> From: Ted Yu 
>> >> Sent: Thursday, September 22, 2016 2:00 PM
>> >> To: dev@hbase.apache.org
>> >> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
>> >>
>> >> I agree - backup / restore is in the same category as import / export.
>> >>
>> >> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
>> andrew.purt...@gmail.com>
>> >> wrote:
>> >>
>> >>> Backup is extra tooling around core in my opinion. Like import or
>> export.
>> >>> Or the optional MOB tool. It's fine.
>> >>>
>> >>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi 
>> >>> wrote:
>> >>>>
>> >>>> What's the latest opinion around running MR jobs from hbase (Master
>> or
>> >>> RS)?
>> >>>>
>> >>>> I remember in the past that there was discussion about not having MR
>> >> has
>> >>>> direct dependency of hbase.
>> >>>>
>> >>>> I think some of discussion where around MOB that had a MR job to
>> >> compact,
>> >>>> that later was transformed in a non-MR job to be merged, I think we
>> >> had a
>> >>>> similar discussion for log split/replay.
>> >>>>
>> >>>> the latest is the new Backup feature (HBASE-7912), that runs a MR job
>> >>> from
>> >>>> the master to copy data or restore data.
>> >>>> (backup is also "not really core" as in.. if you don't use backup
>> >> you'll
>> >>>> not end up running MR jobs, but this was probably true for MOB as in
>> >> "if
>> >>>> you don't enable MOB you don't need MR")
>> >>>>
>> >>>> any thoughts? do we a rule that says "we don't want to have hbase run
>> >> MR
>> >>>> jobs, only tool started manually by the user can do that". or can we
>> >>> start
>> >>>> adding MR calls around without problems?
>> >>>
>> >>
>>
>
>

Re: [DISCUSSION] MR jobs started by Master or RS

[张铎]

I‘m -1 on let master or rs launch MR jobs. It is OK that some of our
features depend on MR but I think the bottom line is that we should launch
the jobs from outside manually or by other services.

2016-09-23 9:47 GMT+08:00 Andrew Purtell :

> Ok, got it. Well "shelling out" is on the line I think, so a fair question.
>
> Can this be driven by a utility derived from Tool like our other MR apps?
> The issue is needing the AccessController to decide if allowed? But nothing
> prevents the user from running the job manually/independently, right?
>
> > On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi 
> wrote:
> >
> > just a remark. my query was not about tools using MR (everyone i think is
> > ok with those).
> > the topic was about: "are we ok with running MR jobs from Master and RSs
> > code?" since this will be the first time we do this
> >
> > Matteo
> >
> >
> >> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das 
> wrote:
> >>
> >> Very much agree; for tools like ExportSnapshot / Backup / Restore, it's
> >> fine to be dependent on MR. MR is the right framework for such. We
> should
> >> also do compactions using MR (just saying :) )
> >> 
> >> From: Ted Yu 
> >> Sent: Thursday, September 22, 2016 2:00 PM
> >> To: dev@hbase.apache.org
> >> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> >>
> >> I agree - backup / restore is in the same category as import / export.
> >>
> >> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
> andrew.purt...@gmail.com>
> >> wrote:
> >>
> >>> Backup is extra tooling around core in my opinion. Like import or
> export.
> >>> Or the optional MOB tool. It's fine.
> >>>
>  On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi 
> >>> wrote:
> 
>  What's the latest opinion around running MR jobs from hbase (Master or
> >>> RS)?
> 
>  I remember in the past that there was discussion about not having MR
> >> has
>  direct dependency of hbase.
> 
>  I think some of discussion where around MOB that had a MR job to
> >> compact,
>  that later was transformed in a non-MR job to be merged, I think we
> >> had a
>  similar discussion for log split/replay.
> 
>  the latest is the new Backup feature (HBASE-7912), that runs a MR job
> >>> from
>  the master to copy data or restore data.
>  (backup is also "not really core" as in.. if you don't use backup
> >> you'll
>  not end up running MR jobs, but this was probably true for MOB as in
> >> "if
>  you don't enable MOB you don't need MR")
> 
>  any thoughts? do we a rule that says "we don't want to have hbase run
> >> MR
>  jobs, only tool started manually by the user can do that". or can we
> >>> start
>  adding MR calls around without problems?
> >>>
> >>
>

Re: Failure: HBase Generate Website

[张铎]

Ah we also need to use java8 for the jenkins jobs...

Thanks for quick action.

2016-09-12 22:32 GMT+08:00 Dima Spivak :

> Looks like our new Java 8 requirement for master broke this one. I'll fix
> it shortly...
>
> On Monday, September 12, 2016, Apache Jenkins Server <
> jenk...@builds.apache.org> wrote:
>
> > Build status: Failure
> >
> > If successful, the website and docs have been generated. To update the
> > live site, follow the instructions below. If failed, skip to the bottom
> of
> > this email.
> >
> > Use the following commands to download the patch and apply it to a clean
> > branch based on origin/asf-site. If you prefer to keep the hbase-site
> repo
> > around permanently, you can skip the clone step.
> >
> >   git clone https://git-wip-us.apache.org/repos/asf/hbase-site.git
> >
> >   cd hbase-site
> >   wget -O- https://builds.apache.org/job/hbase_generate_website/340/
> > artifact/website.patch.zip | funzip > ${GIT_SHA}.patch
> >   git fetch
> >   git checkout -b asf-site-${GIT_SHA} origin/asf-site
> >   git am --whitespace=fix $GIT_SHA.patch
> >
> > At this point, you can preview the changes by opening index.html or any
> of
> > the other HTML pages in your local asf-site-${GIT_SHA} branch.
> >
> > There are lots of spurious changes, such as timestamps and CSS styles in
> > tables, so a generic git diff is not very useful. To see a list of files
> > that have been added, deleted, renamed, changed type, or are otherwise
> > interesting, use the following command:
> >
> >   git diff --name-status --diff-filter=ADCRTXUB origin/asf-site
> >
> > To see only files that had 100 or more lines changed:
> >
> >   git diff --stat origin/asf-site | grep -E '[1-9][0-9]{2,}'
> >
> > When you are satisfied, publish your changes to origin/asf-site using
> > these commands:
> >
> >   git commit --allow-empty -m "Empty commit" # to work around a current
> > ASF INFRA bug
> >   git push origin asf-site-${GIT_SHA}:asf-site
> >   git checkout asf-site
> >   git branch -D asf-site-${GIT_SHA}
> >
> > Changes take a couple of minutes to be propagated. You can verify whether
> > they have been propagated by looking at the Last Published date at the
> > bottom of http://hbase.apache.org/. It should match the date in the
> > index.html on the asf-site branch in Git.
> >
> > As a courtesy- reply-all to this email to let other committers know you
> > pushed the site.
> >
> >
> >
> > If failed, see https://builds.apache.org/job/hbase_generate_website/340/
> > console
>
>
>
> --
> -Dima
>

Re: [DISCUSS] Drop the support of jdk7 at a future 1.x release

[张铎]

Yeah I'm currently working on HBASE-16591 to bring our master build
pipeline to jdk8 only...

2016-09-10 2:11 GMT+08:00 Josh Elser :

>
>
> Sean Busbey wrote:
>
>> This would be a very big breaking change in release numbering that goes
>> against our compatibility guidelines. We only drop support for Java
>> versions on major releases.
>>
>> If we want features that require newer jdks sooner, we should make major
>> releases sooner.
>>
>
> +1 On this exactly
>
>
> On Sep 8, 2016 20:54, "Duo Zhang"  wrote:
>>
>> The main reason is the asynchronous api we want to introduce in HBase
>>> today. See HBASE-13784 and HBASE-16505.
>>>
>>> The CompletableFuture in java8 is very suitable to use as the return
>>> value
>>> of a async method. We can not use it if we still want to support java7,
>>> and
>>> sadly, there is no candidate which is good enough to replace
>>> CompletableFuture. ListenableFuture in guava or Promise in netty are
>>> good,
>>> but we do not want to expose third-party classes in our public
>>> API(especially guava, you know...). And we can also implement our own
>>> ListenableFuture but it just a copy of guava. Or introduce a simple
>>> Callback interface which does not need much code(for us) but this is a
>>> code
>>> style around 2000s so people will not like it...
>>>
>>> And luckily, I found that in our documentation
>>>
>>> http://hbase.apache.org/book.html#basic.prerequisites
>>>
>>> We only say that 1.3 will be compatible with jdk7, not all 1.x.
>>>
>>> So here I propose that we drop the support of jdk7 in a future 1.x
>>> release,
>>> maybe 1.4? Thus we can use CompletableFuture in both master and branch-1.
>>>
>>> Thanks.
>>>
>>>
>>

Re: [ANNOUNCE] Misty Stanley-Jones joins the Apache HBase PMC

[张铎]

Congratulations!

2016-09-08 7:56 GMT+08:00 Jonathan Hsieh :

> Congrats Misty!
>
> On Wed, Sep 7, 2016 at 11:40 AM, Sean Busbey  wrote:
>
> > On behalf of the Apache HBase PMC I am pleased to announce that Misty has
> > accepted our invitation to become a PMC member on the Apache HBase
> project.
> > Misty has been a committer for almost 2 years now. She's done a great job
> > of steering us good directions both for docs and as a project.
> >
> > Please join me in welcoming Misty to the HBase PMC
> >
> > -busbey
> >
>
>
>
> --
> // Jonathan Hsieh (shay)
> // HBase Tech Lead, Software Engineer, Cloudera
> // j...@cloudera.com // @jmhsieh
>

Re: 答复: [ANNOUNCE] Duo Zhang (张铎) joins the Apache HBase PMC

[张铎]

Thanks all. Will do my best.

2016-09-07 13:58 GMT+08:00 Honghua Feng 冯宏华 :

> Congratulations, Duo.
> 
> 发件人: Ted Yu 
> 发送时间: 2016年9月7日 12:31
> 收件人: u...@hbase.apache.org
> 抄送: HBase Dev List
> 主题: Re: [ANNOUNCE] Duo Zhang (张铎) joins the Apache HBase PMC
>
> Congratulations, Duo.
>
> > On Sep 6, 2016, at 9:26 PM, Stack  wrote:
> >
> > On behalf of the Apache HBase PMC I am pleased to announce that 张铎
> > has accepted our invitation to become a PMC member on the Apache
> > HBase project. Duo has healthy notions on where the project should be
> > headed and over the last year and more has been working furiously to take
> > us there.
> >
> > Please join me in welcoming Duo to the HBase PMC!
> >
> > One of us!
> > St.Ack
>

Re: Source incompatibility between 1.2.[0,1] and 1.2.2 breaks Phoenix

[张铎]

Yes, with jdk8 we could add new methods in interface without breaking the
sub classes.

So let's revert HBASE-15645 from branch-1 and branch-1,x and add default
implementations in master? Is yetus OK to run pre-commit with jdk8 only on
master now?

Thanks.

2016-08-16 13:12 GMT+08:00 Sean Busbey :

> On Tue, Aug 16, 2016 at 12:09 AM, Sean Busbey  wrote:
>
> >
> > Moving from interfaces to abstract classes requires a deprecation cycle,
> > so to transition I think we'd need to introduce a new API. Anyone up for
> > this in 2.0?
> >
>
> I just remembered that HBase 2.0 is moving to jdk8+, which means we could
> use default methods when adding to existing interfaces, so there's
> substantially less draw for moving to abstract classes.
>

Re: [DISCUSS] HBase-2.0 SHOULD be rolling upgradable and wire-compatible with 1.x

[张铎]

+1 on requiring upgrading to one of the latest 1.x.y before upgrading to
2.0.

2016-06-21 14:30 GMT+08:00 Matteo Bertozzi :

> I think everyone wants rolling upgrade. the discussion should probably be
> around how much compatibility code do we want to keep around.
>
> using as example HBASE-16060, we need to decide how much are we rolling
> upgradable and from where.
> I'm not too convinced that we should have extra code in master to "simulate
> the old states",
> I'll rather have cleaner code in 2.0 and force the users to move to one of
> the latest 1.x.y
> there are not many changes in the 1.x releases, so we should be able to
> say:
> if you are on 1.1 move to the latest 1.1.x, if you are on 1.2 move to the
> latest 1.2.x and so on.
>
> also there are some operations that may not be needed during rolling
> upgrades,
> and we can cut on compatibility to have some code removed.
> an example here is HBASE-15521 where we are no longer able to clone/restore
> snapshot during 1.x -> 2.x rolling upgrade, until the two master are on
> 2.x. but this may be extended to you can't perform some operation until all
> the machines are on 2.x for some future change.
>
> I think we should aim for something like:
>  - data path: HTable put/get/scan/... must work during a rolling upgrade
>  - replication: must? work during rolling upgrade
>  - admin: some operation may not be working during rolling upgrade
>  - upgrade to the latest 1.x.y before the 2.x upgrade (we can add in 2.x
> master and rs the ability to check the client version)
>
>
> Matteo
>
>
> On Tue, Jun 21, 2016 at 12:05 AM, Dima Spivak 
> wrote:
>
> > If there’s no technical limitation, we should definitely do it. As you
> > note, customers running in production hate when they have to shut down
> > clusters and with some of the testing infrastructure being rolled out,
> this
> > is definitely something we can set up automated testing for. +1
> >
> > -Dima
> >
> > On Mon, Jun 20, 2016 at 2:58 PM, Enis Söztutar  wrote:
> >
> > > Time to formalize 2.0 rolling upgrade scenario?
> > >
> > > 0.94 -> 0.96 singularity was a real pain for operators and for our
> users.
> > > If possible we should not have the users suffer through the same thing
> > > unless there is a very compelling reason. For the current stuff in
> > master,
> > > there is nothing that will prevent us to not have rolling upgrade
> support
> > > for 2.0. So I say, we should decide on the rolling upgrade requirement
> > now,
> > > and start to evaluate incoming patches accordingly. Otherwise, we risk
> > the
> > > option to go deeper down the hole.
> > >
> > > What do you guys think. Previous threads [1] and [2] seems to be in
> > favor.
> > > Should we vote?
> > >
> > > Ref:
> > > [1]
> > >
> > >
> >
> http://search-hadoop.com/m/YGbbsd4An1aso5E1&subj=HBase+1+x+to+2+0+upgrade+goals+
> > >
> > > [2]
> > >
> > >
> >
> http://search-hadoop.com/m/YGbb1CBXTL8BTI&subj=thinking+about+supporting+upgrades+to+HBase+1+x+and+2+x
> > >
> >
>

Re: [DISCUSS] Make AsyncFSWAL the default WAL in 2.0

[张铎]

I think at that time I will start a new project called AsyncDFSClient which
will implement the whole client side logic of HDFS without using reflection
:)

2016-05-12 10:27 GMT+08:00 Andrew Purtell :

> If Hadoop refuses the changes before we release, we can change the default
> back.
>
>
> On May 11, 2016, at 6:50 PM, Gary Helmling  wrote:
>
> >>
> >>
> >> I was trying to avoid the below oft-repeated pattern at least for the
> case
> >> of critical developments:
> >>
> >> + New feature arrives after much work by developer, reviewers and
> testers
> >> accompanied by fanfare (blog, talks).
> >> + Developers and reviewers move on after getting it committed or it gets
> >> hacked into a deploy so it works in a frankenstein form
> >> + It sits in our code base across one or more releases marked as
> optional,
> >> 'experimental'
> >> + The 'experimental' bleamish discourages its exercise by users
> >> + The feature lags, rots
> >> + Or, the odd time, we go ahead and enable it as default in spite of the
> >> fact it was never tried when experimental.
> >>
> >> Distributed Log Replay sat in hbase across a few major versions. Only
> when
> >> the threat of our making an actual release with it on by default did it
> get
> >> serious attention where it was found flawed and is now being actively
> >> purged. This was after it made it past reviews, multiple attempts at
> >> testing at scale, and so on; i.e. we'd done it all by the book. The
> time in
> >> an 'experimental' state added nothing.
> > Those are all valid concerns as well. It's certainly a pattern that we've
> > seen repeated. That's also a broader concern I have about the farther we
> > push out 2.0, then the less exercised master is.
> >
> > I don't really know how best to balance this with concerns about user
> > stability.  Enabling by default in master would certainly be a forcing
> > function and would help it get more testing before release.  I hear that
> > argument.  But I'm worried about the impact after release, where
> something
> > as simple as a bug-fix point release upgrade of Hadoop could result in
> > runtime breakage of an HBase install.  Will this happen in practice?  I
> > don't know.  It seems unlikely that the private variable names being used
> > for example would change in a point release.  But we're violating the
> > abstraction that Hadoop provides us which guarantees such breakage won't
> > occur.
> >
> >
> >>> Yes. 2.0 is a bit out there so we have some time to iron out issues is
> >> the
> >> thought. Yes, it could push out delivery of 2.0.
> > Having this on by default in an unreleased master doesn't actually worry
> me
> > that much.  It's just the question of what happens when we do release.
> At
> > that point, this discussion will be ancient history and I don't think
> we'll
> > give any renewed consideration to what the impact of this change might
> be.
> > Ideally it would be great to see this work in HDFS by that point and for
> > that HDFS version this becomes a non-issue.
> >
> >
> >>
> >> I think the discussion here has been helpful. Holes have been found (and
> >> plugged), the risk involved has gotten a good airing out here on dev,
> and
> >> in spite of the back and forth, one of our experts in good standing is
> >> still against it being on by default.
> >>
> >> If you are not down w/ the arguments, I'd be fine not making it the
> >> default.
> >> St.Ack
> >
> > I don't think it's right to block this by myself, since I'm clearly in
> the
> > minority.  Since others clearly support this change, have at it.
> >
> > But let me pose an alternate question: what if HDFS flat out refuses to
> > adopt this change?  What are our options then with this already shipping
> as
> > a default?  Would we continue to endure breakage due to the use of HDFS
> > private internals?  Do we switch the default back?  Do we do something
> else?
> >
> > Thanks for the discussion.
>

Re: [DISCUSS] Make AsyncFSWAL the default WAL in 2.0

[张铎]

See HDFS-223 and HDFS-916. There are plenty of issues related. The most
important thing is that we need a suitable api and there is an asynchronous
file system proposal in HADOOP-12910 which does not fit our requirements so
I need to stop it being committed first...

And a default choice in a later HBase version means that this feature is
definitely needed in HDFS and will be well tested even before it go into
the HDFS code base. An experimental feature is always experimental. Think
of DLR. For per CF flush, the data loss issue had been found before
releasing 1.2 in which version it is on by default. And we also experienced
this recently when backporting the scan heartbeat feature into our
internal branch. Lots of small bugs though the code has been there for
months...

Thanks.

2016年5月11日星期三，Gary Helmling  写道：

> >
> > Yeah the 'push to upstream' work has been started already. See here
> >
> > https://issues.apache.org/jira/browse/HADOOP-12910
> >
> > But it is much harder to push code into HDFS than HBase. It is the core
> of
> > all hadoop systems and I do not have many contacts in the hdfs
> community...
> >
> >
> Yes, I'm familiar with the difficulty of getting even relatively small
> change in to HDFS.
>
> Is HADOOP-12910 really the right issue for this?  There is some good
> discussion there, but it looks like it's primarily motivated by doing large
> batches of async renames.  Isn't our issue more that we want a whole new
> OutputStream implementation doing fan out instead of the regular pipeline
> replication?
>
> HDFS-9924 seems like it might be the better umbrella for this.  Maybe it
> would be better to create a new top level issue and link it there and
> comment in HDFS-9924 to try to get some traction.
>
>
> > And it is more convincing if we make it default as it means that we will
> > keep maintaining the code rather than make it stale and unstable.
> >
> >
> I would agree with this reasoning if we were talking about making an
> implementation inside HDFS the default.  That would indicate commitment to
> contribute to and maintain the HDFS implementation.  Making a separate code
> copy living in HBase the default I don't think really means anything for
> HDFS.
>
> The fact that this already needs to be updated for 2.8 just reinforces that
> we're going to see maintainability issues with this.
>
> Again, I appreciate all of the work that has gone in to this feature and
> the big performance improvements it shows, but I think the sequencing of
> events here is going to ultimately cause us and our users more pain than is
> necessary.
>

Re: [DISCUSS] Make AsyncFSWAL the default WAL in 2.0

[张铎]

Some methods are moved from classes in hadoop-hdfs to classes in
hadoop-hdfs-client.
ClientProtocol.addBlock method adds an extra parameter.
DFSClient.Conf is moved to a separated file and renamed to DFSClientConf.

Not very hard. I promise that I can give a patch within 3 days after the
release of hadoop-2.8.0.

Thanks.

2016-05-10 16:23 GMT+08:00 张铎 :

> Yeah the 'push to upstream' work has been started already. See here
>
> https://issues.apache.org/jira/browse/HADOOP-12910
>
> But it is much harder to push code into HDFS than HBase. It is the core of
> all hadoop systems and I do not have many contacts in the hdfs community...
>
> And it is more convincing if we make it default as it means that we will
> keep maintaining the code rather than make it stale and unstable.
>
> And for the compatibility of different hadoop versions, I think I can deal
> with it. Let me check hadoop-2.8.0-SNAPSHOT first.
>
> Thanks.
>
> 2016-05-10 14:59 GMT+08:00 Gary Helmling :
>
>> Thanks for adding the tests and fixing up AES support.
>>
>> My only real concern is the maintainability of this code as our own
>> private
>> DFS client.  The SASL support, for example, is largely based on reflection
>> and reaches in to private fields of @InterfaceAudience.Private Hadoop
>> classes.  This seems bound to break with a future Hadoop release.  I
>> appreciate the parameterized testing wrapped around this because it
>> doesn't
>> seem like we'll have much else in the way of safety checking.  This is not
>> a knock on the code -- it's a pretty clean reach into the HDFS guts, but a
>> reach it is.  For a component at the core of our data integrity, this
>> seems
>> like a risk.
>>
>> To me, it seems much safer to actively try to push this upstream into HDFS
>> right now, and still pointing to its optional, non-default use in HBase as
>> a compelling story.  I don't understand why making it the default in 2.0
>> is
>> necessary for this.  Do you really think it will make that big a
>> difference
>> for upstreaming?  Once it's actually in Hadoop and maintained, it seems
>> like a no-brainer to make it the default.
>>
>> On Mon, May 9, 2016 at 5:09 PM Stack  wrote:
>>
>> > Any other suggestions/objections here? If not, will make the cut over in
>> > next day or so.
>> > Thanks,
>> > St.Ack
>> >
>> > On Thu, May 5, 2016 at 10:02 PM, Stack  wrote:
>> >
>> > > On Thu, May 5, 2016 at 7:39 PM, Yu Li  wrote:
>> > >
>> > >> Almost miss the party...
>> > >>
>> > >> bq. Do you think it worth to backport this feature to branch-1 and
>> > release
>> > >> it in the next 1.x release? This may introduce a compatibility issue
>> as
>> > >> said
>> > >> in HBASE-14949 that we need HBASE-14949 to make sure that the rolling
>> > >> upgrade
>> > >> does not lose data...
>> > >> From current perf data I think the effort is worthwhile, we already
>> > >> started
>> > >> some work here and will run it on production after some carefully
>> > testing
>> > >> (and of course, if the perf number confirmed, but I'm optimistic
>> somehow
>> > >> :-P). Regarding HBASE-14949, I guess a two-step rolling upgrade will
>> > make
>> > >> it work, right? (And I guess this will also be a question when we
>> > upgrade
>> > >> from 1.x to 2.0 later?)
>> > >>
>> > >>
>> > > Or a clean shutdown and restart? Or a fresh install? I'd think
>> backport
>> > > would be fine if you have to enable it and it has warnings and is
>> clear
>> > on
>> > > circumstances under which there could be dataloss.
>> > >
>> > > St.Ack
>> > >
>> > >
>> > >
>> > >> btw, I'm +1 about making asyncfswal as default in 2.0 :-)
>> > >>
>> > >> Best Regards,
>> > >> Yu
>> > >>
>> > >> On 6 May 2016 at 09:49, Ted Yu  wrote:
>> > >>
>> > >> > Thanks for your effort, Duo.
>> > >> >
>> > >> > I am in favor of turning AsyncWAL as default in master branch.
>> > >> >
>> > >> > Cheers
>> > >> >
>> > >> > On Thu, May 5, 2016 at 6:03 PM, 张铎  wrote:
>&g

Re: [DISCUSS] Make AsyncFSWAL the default WAL in 2.0

[张铎]

Yeah the 'push to upstream' work has been started already. See here

https://issues.apache.org/jira/browse/HADOOP-12910

But it is much harder to push code into HDFS than HBase. It is the core of
all hadoop systems and I do not have many contacts in the hdfs community...

And it is more convincing if we make it default as it means that we will
keep maintaining the code rather than make it stale and unstable.

And for the compatibility of different hadoop versions, I think I can deal
with it. Let me check hadoop-2.8.0-SNAPSHOT first.

Thanks.

2016-05-10 14:59 GMT+08:00 Gary Helmling :

> Thanks for adding the tests and fixing up AES support.
>
> My only real concern is the maintainability of this code as our own private
> DFS client.  The SASL support, for example, is largely based on reflection
> and reaches in to private fields of @InterfaceAudience.Private Hadoop
> classes.  This seems bound to break with a future Hadoop release.  I
> appreciate the parameterized testing wrapped around this because it doesn't
> seem like we'll have much else in the way of safety checking.  This is not
> a knock on the code -- it's a pretty clean reach into the HDFS guts, but a
> reach it is.  For a component at the core of our data integrity, this seems
> like a risk.
>
> To me, it seems much safer to actively try to push this upstream into HDFS
> right now, and still pointing to its optional, non-default use in HBase as
> a compelling story.  I don't understand why making it the default in 2.0 is
> necessary for this.  Do you really think it will make that big a difference
> for upstreaming?  Once it's actually in Hadoop and maintained, it seems
> like a no-brainer to make it the default.
>
> On Mon, May 9, 2016 at 5:09 PM Stack  wrote:
>
> > Any other suggestions/objections here? If not, will make the cut over in
> > next day or so.
> > Thanks,
> > St.Ack
> >
> > On Thu, May 5, 2016 at 10:02 PM, Stack  wrote:
> >
> > > On Thu, May 5, 2016 at 7:39 PM, Yu Li  wrote:
> > >
> > >> Almost miss the party...
> > >>
> > >> bq. Do you think it worth to backport this feature to branch-1 and
> > release
> > >> it in the next 1.x release? This may introduce a compatibility issue
> as
> > >> said
> > >> in HBASE-14949 that we need HBASE-14949 to make sure that the rolling
> > >> upgrade
> > >> does not lose data...
> > >> From current perf data I think the effort is worthwhile, we already
> > >> started
> > >> some work here and will run it on production after some carefully
> > testing
> > >> (and of course, if the perf number confirmed, but I'm optimistic
> somehow
> > >> :-P). Regarding HBASE-14949, I guess a two-step rolling upgrade will
> > make
> > >> it work, right? (And I guess this will also be a question when we
> > upgrade
> > >> from 1.x to 2.0 later?)
> > >>
> > >>
> > > Or a clean shutdown and restart? Or a fresh install? I'd think backport
> > > would be fine if you have to enable it and it has warnings and is clear
> > on
> > > circumstances under which there could be dataloss.
> > >
> > > St.Ack
> > >
> > >
> > >
> > >> btw, I'm +1 about making asyncfswal as default in 2.0 :-)
> > >>
> > >> Best Regards,
> > >> Yu
> > >>
> > >> On 6 May 2016 at 09:49, Ted Yu  wrote:
> > >>
> > >> > Thanks for your effort, Duo.
> > >> >
> > >> > I am in favor of turning AsyncWAL as default in master branch.
> > >> >
> > >> > Cheers
> > >> >
> > >> > On Thu, May 5, 2016 at 6:03 PM, 张铎  wrote:
> > >> >
> > >> > > Some progress.
> > >> > >
> > >> > > I have filed HBASE-15743 for the transparent encryption support,
> > >> > > and HBASE-15754 for the AES encryption UT. Now both of them are
> > >> resolved.
> > >> > > Let's resume the discussion here.
> > >> > >
> > >> > > Thanks.
> > >> > >
> > >> > > 2016-05-03 10:09 GMT+08:00 张铎 :
> > >> > >
> > >> > > > Fine, will add the testcase.
> > >> > > >
> > >> > > > And for the RPC, we only implement a new client side DTP here
> and
> > >> still
> > >> > > > use the original RPC.
> > >> > > >
>

Re: [DISCUSS] Make AsyncFSWAL the default WAL in 2.0

[张铎]

Some progress.

I have filed HBASE-15743 for the transparent encryption support,
and HBASE-15754 for the AES encryption UT. Now both of them are resolved.
Let's resume the discussion here.

Thanks.

2016-05-03 10:09 GMT+08:00 张铎 :

> Fine, will add the testcase.
>
> And for the RPC, we only implement a new client side DTP here and still
> use the original RPC.
>
> Thanks.
>
> 2016-05-03 3:20 GMT+08:00 Gary Helmling :
>
>> On Fri, Apr 29, 2016 at 6:24 PM 张铎  wrote:
>>
>> > Yes, it does. There is testcase that enumerates all the possible
>> protection
>> > level(authentication, integrity and privacy) and encryption
>> algorithm(none,
>> > 3des, rc4).
>> >
>> >
>> >
>> https://github.com/apache/hbase/blob/master/hbase-server/src/test/java/org/apache/hadoop/hbase/io/asyncfs/TestSaslFanOutOneBlockAsyncDFSOutput.java
>> >
>> > I have also tested it in a secure cluster(hbase-2.0.0-SNAPSHOT and
>> > hadoop-2.4.0).
>> >
>>
>> Thanks.  Can you add in support for testing with AES
>> (dfs.encrypt.data.transfer.cipher.suites=AES/CTR/NoPadding)?  This is only
>> available in Hadoop 2.6.0+, but I think is far more likely to be used in
>> production than 3des or rc4.
>
>
>> Also, have you been following HADOOP-10768?  That is changing Hadoop RPC
>> encryption negotiation to support more performant AES wrapping, similar to
>> what is now supported in the data transfer pipeline.
>>
>
>

Re: [DISCUSS] Make AsyncFSWAL the default WAL in 2.0

[张铎]

Fine, will add the testcase.

And for the RPC, we only implement a new client side DTP here and still use
the original RPC.

Thanks.

2016-05-03 3:20 GMT+08:00 Gary Helmling :

> On Fri, Apr 29, 2016 at 6:24 PM 张铎  wrote:
>
> > Yes, it does. There is testcase that enumerates all the possible
> protection
> > level(authentication, integrity and privacy) and encryption
> algorithm(none,
> > 3des, rc4).
> >
> >
> >
> https://github.com/apache/hbase/blob/master/hbase-server/src/test/java/org/apache/hadoop/hbase/io/asyncfs/TestSaslFanOutOneBlockAsyncDFSOutput.java
> >
> > I have also tested it in a secure cluster(hbase-2.0.0-SNAPSHOT and
> > hadoop-2.4.0).
> >
>
> Thanks.  Can you add in support for testing with AES
> (dfs.encrypt.data.transfer.cipher.suites=AES/CTR/NoPadding)?  This is only
> available in Hadoop 2.6.0+, but I think is far more likely to be used in
> production than 3des or rc4.


> Also, have you been following HADOOP-10768?  That is changing Hadoop RPC
> encryption negotiation to support more performant AES wrapping, similar to
> what is now supported in the data transfer pipeline.
>

Re: [DISCUSS] Make AsyncFSWAL the default WAL in 2.0

[张铎]

I checked the code. The HdfsFileStatus returned when creating of a file
inside encryption zone will contain a FileEncryptionInfo. DFSClient will
create a CryptoOutputStream which wraps a DFSOutputStream based on the
FileEntryptionInfo.

Let me file issue to implement it. Just another piece of reflection codes...

Thanks.

2016-05-01 13:06 GMT+08:00 Sean Busbey :

> On Sat, Apr 30, 2016 at 1:34 PM, Stack  wrote:
> > On Sat, Apr 30, 2016 at 6:33 AM, Ted Yu  wrote:
> >
> >> What about support for Transparent Data Encryption feature which was
> >> introduced in Apache Hadoop 2.6.0 ?
> >>
> >>
> > Transparent: "...(of a process or interface) functioning without the user
> > being aware of its presence."
> > St.Ack
> >
> >
> >
>
> It's only transparent when the application is making use of the
> libraries provided by the project. ;)
>
> HDFS Transparent Encryption is all on the client side:
>
>
> http://hadoop.apache.org/docs/r2.6.4/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html
>

Re: [DISCUSS] Make AsyncFSWAL the default WAL in 2.0

[张铎]

Yes, it does. There is testcase that enumerates all the possible protection
level(authentication, integrity and privacy) and encryption algorithm(none,
3des, rc4).

https://github.com/apache/hbase/blob/master/hbase-server/src/test/java/org/apache/hadoop/hbase/io/asyncfs/TestSaslFanOutOneBlockAsyncDFSOutput.java

I have also tested it in a secure cluster(hbase-2.0.0-SNAPSHOT and
hadoop-2.4.0).

Thanks.

2016-04-30 2:32 GMT+08:00 Gary Helmling :

> How well has this been tested on secure clusters?  I know SASL support was
> lacking initially, but I believe it had been added?  Does AsyncFSWAL
> support all the HDFS transport encryption options?
>
>
> On Fri, Apr 29, 2016 at 12:05 AM Stack  wrote:
>
> > I'm +1 on enabling asyncfswal as default in 2.0:
> >
> > + We'll have plenty of time to figure issues if any if we get it in now,
> > early.
> > + The improvement in throughput is substantial
> > + There are now less moving parts
> > + A critical piece of our write path is much less opaque in its workings
> > and no longer (effectively) immutable
> >
> > St.Ack
> >
> >
> > On Thu, Apr 28, 2016 at 11:53 PM, 张铎  wrote:
> >
> > > I‘ve done dig in HDFS and HADOOP proejcts and found that there is an
> > active
> > > issue HADOOP-12910 that related to asynchronous FileSystem
> > implementation.
> > >
> > > I have left some comments on it, maybe we could start from there.
> > >
> > > Thanks.
> > >
> > > 2016-04-29 14:42 GMT+08:00 Stack :
> > >
> > > > On Thu, Apr 28, 2016 at 8:47 PM, Ted Yu  wrote:
> > > >
> > > > > Last comment on HDFS-916 was from 2010.
> > > > >
> > > > > Suggest making a new issue or reviving discussion on HDFS-916
> > > (currently
> > > > > assigned to Todd).
> > > > >
> > > > >
> > > > Duo is on it. Some mileage and confidence in the new code would be
> good
> > > to
> > > > have before going to HDFS (Getting stuff into HDFS is a PITA at the
> > best
> > > of
> > > > times... lets have a good case when we go there).
> > > >
> > > >
> > > > > bq. The fallback implementation is not aim to get a good
> performance
> > > > >
> > > > > For more than two weeks, I have been working with Azure Data Lake
> > > > > developers so that all hbase system tests pass on ADLS - there were
> > > > subtle
> > > > > differences between ADLS and hdfs.
> > > > >
> > > > > If switching to AsyncWAL gives either WASB or ADLS subpar
> > performance,
> > > it
> > > > > would make upgrading to hbase 2.x unacceptable for their users.
> > > > >
> > > > >
> > > > Just use FSHLog instead of asyncfswal when up on WASB. Its just a
> > config
> > > > change.
> > > >
> > > > St.Ack
> > > >
> > > >
> > > >
> > > > > On Thu, Apr 28, 2016 at 8:39 PM, 张铎  wrote:
> > > > >
> > > > > > 2016-04-29 11:35 GMT+08:00 Ted Yu :
> > > > > >
> > > > > > > bq. AsyncFSOutput will be in HDFS-3.0
> > > > > > >
> > > > > > > Is there HDFS JIRA for the above ? Can you share the number ?
> > > > > > >
> > > > > > I have not filed a new one but there are bunch of related issues
> > > > already,
> > > > > > such as this one https://issues.apache.org/jira/browse/HDFS-916
> > > > > >
> > > > > > >
> > > > > > > bq. Just wrap FSDataOutputStream to make it act like an
> > > asynchronous
> > > > > > output
> > > > > > >
> > > > > > > Can you be a bit more specific ?
> > > > > > > HBase currently works with WASB and Azure Data Lake. Does the
> > above
> > > > > mean
> > > > > > > their performance would suffer ?
> > > > > > >
> > > > > > Yes, the performance will suffer...
> > > > > > The fallback implementation is not aim to get a good performance,
> > > just
> > > > > for
> > > > > > compatibility with any FileSystem implementation.
> > > > > >
> > > > > > >
> > > > > > > On Thu, Apr 28, 2016 at 8:30 PM,

Re: [DISCUSS] Make AsyncFSWAL the default WAL in 2.0

[张铎]

I‘ve done dig in HDFS and HADOOP proejcts and found that there is an active
issue HADOOP-12910 that related to asynchronous FileSystem implementation.

I have left some comments on it, maybe we could start from there.

Thanks.

2016-04-29 14:42 GMT+08:00 Stack :

> On Thu, Apr 28, 2016 at 8:47 PM, Ted Yu  wrote:
>
> > Last comment on HDFS-916 was from 2010.
> >
> > Suggest making a new issue or reviving discussion on HDFS-916 (currently
> > assigned to Todd).
> >
> >
> Duo is on it. Some mileage and confidence in the new code would be good to
> have before going to HDFS (Getting stuff into HDFS is a PITA at the best of
> times... lets have a good case when we go there).
>
>
> > bq. The fallback implementation is not aim to get a good performance
> >
> > For more than two weeks, I have been working with Azure Data Lake
> > developers so that all hbase system tests pass on ADLS - there were
> subtle
> > differences between ADLS and hdfs.
> >
> > If switching to AsyncWAL gives either WASB or ADLS subpar performance, it
> > would make upgrading to hbase 2.x unacceptable for their users.
> >
> >
> Just use FSHLog instead of asyncfswal when up on WASB. Its just a config
> change.
>
> St.Ack
>
>
>
> > On Thu, Apr 28, 2016 at 8:39 PM, 张铎  wrote:
> >
> > > 2016-04-29 11:35 GMT+08:00 Ted Yu :
> > >
> > > > bq. AsyncFSOutput will be in HDFS-3.0
> > > >
> > > > Is there HDFS JIRA for the above ? Can you share the number ?
> > > >
> > > I have not filed a new one but there are bunch of related issues
> already,
> > > such as this one https://issues.apache.org/jira/browse/HDFS-916
> > >
> > > >
> > > > bq. Just wrap FSDataOutputStream to make it act like an asynchronous
> > > output
> > > >
> > > > Can you be a bit more specific ?
> > > > HBase currently works with WASB and Azure Data Lake. Does the above
> > mean
> > > > their performance would suffer ?
> > > >
> > > Yes, the performance will suffer...
> > > The fallback implementation is not aim to get a good performance, just
> > for
> > > compatibility with any FileSystem implementation.
> > >
> > > >
> > > > On Thu, Apr 28, 2016 at 8:30 PM, 张铎  wrote:
> > > >
> > > > > Inline comments.
> > > > > Thanks,
> > > > >
> > > > > 2016-04-29 10:57 GMT+08:00 Sean Busbey :
> > > > >
> > > > > > I am nervous about having default out-of-the-box new HBase users
> > > > reliant
> > > > > on
> > > > > > a bespoke HDFS client, especially given Hadoop's compatibility
> > > > > > promises and history. Answers for these questions would make me
> > more
> > > > > > confident:
> > > > > >
> > > > > > 1) Where are we on getting the client-side changes to HDFS pushed
> > > back
> > > > > > upstream?
> > > > > >
> > > > > No progress yet... Here I want to tell a good story that HBase is
> > > already
> > > > > use it as default :)
> > > > >
> > > > > >
> > > > > > 2) How well do we detect when our FS is not HDFS and what does
> > > > > > fallback look like?
> > > > > >
> > > > > Just wrap FSDataOutputStream to make it act like an asynchronous
> > > > > output(call hflush in a separated thread). The performance is not
> > good
> > > I
> > > > > think.
> > > > >
> > > > > >
> > > > > > 3) Will this mean altering the versions of Hadoop we label as
> > > > > > supported for HBase 2.y+?
> > > > > >
> > > > > I have tested with hadoop versions from 2.4.x to 2.7.x, so I don't
> > > think
> > > > we
> > > > > need to change the supported versions?
> > > > >
> > > > > >
> > > > > > 4) How are we going to ensure our client remains compatible with
> > > newer
> > > > > > Hadoop releases?
> > > > > >
> > > > > We can not ensure, HDFS always breaks HBase at a new release...
> > > > > I need to test AsyncFSWAL on every new 2.x release and make it
> > > compatible
> > > > > with that version. And back to #1, I think we should make sure that
> > the
> > > > > AsyncFSOutput will be in HDFS-3.0. And in HBase-3.0, we can
> > introduce a
> > > > new
> > > > > 'AsyncFSWAL' that use the AsyncFSOutput in HDFS.
> > > > >
> > > > > >
> > > > > > On Thu, Apr 28, 2016 at 9:42 PM, Duo Zhang 
> > > > wrote:
> > > > > > > Six month after I filed HBASE-14790...
> > > > > > >
> > > > > > > Now the AsyncFSWAL is ready. The WALPE result shows that it is
> > > > > > *1.4x~3.7x*
> > > > > > > faster than FSHLog. The ITBLL result turns out that it is *not
> > bad*
> > > > > than
> > > > > > > FSHLog(the master branch is not that stable itself...).
> > > > > > >
> > > > > > > More details can be found on HBASE-15536.
> > > > > > >
> > > > > > > So here we propose to change the default WAL from FSHLog to
> > > > AsyncFSWAL.
> > > > > > > Suggestions are welcomed.
> > > > > > >
> > > > > > > Thanks.
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > busbey
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Make AsyncFSWAL the default WAL in 2.0

[张铎]

2016-04-29 11:47 GMT+08:00 Ted Yu :

> Last comment on HDFS-916 was from 2010.
>
> Suggest making a new issue or reviving discussion on HDFS-916 (currently
> assigned to Todd).
>
> bq. The fallback implementation is not aim to get a good performance
>
> For more than two weeks, I have been working with Azure Data Lake
> developers so that all hbase system tests pass on ADLS - there were subtle
> differences between ADLS and hdfs.
>
> If switching to AsyncWAL gives either WASB or ADLS subpar performance, it
> would make upgrading to hbase 2.x unacceptable for their users.
>
You can still use FSHLog, it is not removed...
But yes, this is a good point on how we choose default configs in HBase.
A config that performs normally for every case, or a config that performs
much better under the main scenario but worse for other scenarios...

>
> On Thu, Apr 28, 2016 at 8:39 PM, 张铎  wrote:
>
> > 2016-04-29 11:35 GMT+08:00 Ted Yu :
> >
> > > bq. AsyncFSOutput will be in HDFS-3.0
> > >
> > > Is there HDFS JIRA for the above ? Can you share the number ?
> > >
> > I have not filed a new one but there are bunch of related issues already,
> > such as this one https://issues.apache.org/jira/browse/HDFS-916
> >
> > >
> > > bq. Just wrap FSDataOutputStream to make it act like an asynchronous
> > output
> > >
> > > Can you be a bit more specific ?
> > > HBase currently works with WASB and Azure Data Lake. Does the above
> mean
> > > their performance would suffer ?
> > >
> > Yes, the performance will suffer...
> > The fallback implementation is not aim to get a good performance, just
> for
> > compatibility with any FileSystem implementation.
> >
> > >
> > > On Thu, Apr 28, 2016 at 8:30 PM, 张铎  wrote:
> > >
> > > > Inline comments.
> > > > Thanks,
> > > >
> > > > 2016-04-29 10:57 GMT+08:00 Sean Busbey :
> > > >
> > > > > I am nervous about having default out-of-the-box new HBase users
> > > reliant
> > > > on
> > > > > a bespoke HDFS client, especially given Hadoop's compatibility
> > > > > promises and history. Answers for these questions would make me
> more
> > > > > confident:
> > > > >
> > > > > 1) Where are we on getting the client-side changes to HDFS pushed
> > back
> > > > > upstream?
> > > > >
> > > > No progress yet... Here I want to tell a good story that HBase is
> > already
> > > > use it as default :)
> > > >
> > > > >
> > > > > 2) How well do we detect when our FS is not HDFS and what does
> > > > > fallback look like?
> > > > >
> > > > Just wrap FSDataOutputStream to make it act like an asynchronous
> > > > output(call hflush in a separated thread). The performance is not
> good
> > I
> > > > think.
> > > >
> > > > >
> > > > > 3) Will this mean altering the versions of Hadoop we label as
> > > > > supported for HBase 2.y+?
> > > > >
> > > > I have tested with hadoop versions from 2.4.x to 2.7.x, so I don't
> > think
> > > we
> > > > need to change the supported versions?
> > > >
> > > > >
> > > > > 4) How are we going to ensure our client remains compatible with
> > newer
> > > > > Hadoop releases?
> > > > >
> > > > We can not ensure, HDFS always breaks HBase at a new release...
> > > > I need to test AsyncFSWAL on every new 2.x release and make it
> > compatible
> > > > with that version. And back to #1, I think we should make sure that
> the
> > > > AsyncFSOutput will be in HDFS-3.0. And in HBase-3.0, we can
> introduce a
> > > new
> > > > 'AsyncFSWAL' that use the AsyncFSOutput in HDFS.
> > > >
> > > > >
> > > > > On Thu, Apr 28, 2016 at 9:42 PM, Duo Zhang 
> > > wrote:
> > > > > > Six month after I filed HBASE-14790...
> > > > > >
> > > > > > Now the AsyncFSWAL is ready. The WALPE result shows that it is
> > > > > *1.4x~3.7x*
> > > > > > faster than FSHLog. The ITBLL result turns out that it is *not
> bad*
> > > > than
> > > > > > FSHLog(the master branch is not that stable itself...).
> > > > > >
> > > > > > More details can be found on HBASE-15536.
> > > > > >
> > > > > > So here we propose to change the default WAL from FSHLog to
> > > AsyncFSWAL.
> > > > > > Suggestions are welcomed.
> > > > > >
> > > > > > Thanks.
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > busbey
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Make AsyncFSWAL the default WAL in 2.0

[张铎]

But 2.0 is not released yet...

Do you think it worth to backport this feature to branch-1 and release it
in the next 1.x release? This may introduce a compatibility issue as said
in HBASE-14949 that we need HBASE-14949 to make sure that the rolling
upgrade does not lose data...

2016-04-29 11:34 GMT+08:00 Heng Chen :

> The performance is quite great,  but i think maybe we should collect some
> experience on real production cluster before we make it as default.
>
> 2016-04-29 11:30 GMT+08:00 张铎 :
>
> > Inline comments.
> > Thanks,
> >
> > 2016-04-29 10:57 GMT+08:00 Sean Busbey :
> >
> > > I am nervous about having default out-of-the-box new HBase users
> reliant
> > on
> > > a bespoke HDFS client, especially given Hadoop's compatibility
> > > promises and history. Answers for these questions would make me more
> > > confident:
> > >
> > > 1) Where are we on getting the client-side changes to HDFS pushed back
> > > upstream?
> > >
> > No progress yet... Here I want to tell a good story that HBase is already
> > use it as default :)
> >
> > >
> > > 2) How well do we detect when our FS is not HDFS and what does
> > > fallback look like?
> > >
> > Just wrap FSDataOutputStream to make it act like an asynchronous
> > output(call hflush in a separated thread). The performance is not good I
> > think.
> >
> > >
> > > 3) Will this mean altering the versions of Hadoop we label as
> > > supported for HBase 2.y+?
> > >
> > I have tested with hadoop versions from 2.4.x to 2.7.x, so I don't think
> we
> > need to change the supported versions?
> >
> > >
> > > 4) How are we going to ensure our client remains compatible with newer
> > > Hadoop releases?
> > >
> > We can not ensure, HDFS always breaks HBase at a new release...
> > I need to test AsyncFSWAL on every new 2.x release and make it compatible
> > with that version. And back to #1, I think we should make sure that the
> > AsyncFSOutput will be in HDFS-3.0. And in HBase-3.0, we can introduce a
> new
> > 'AsyncFSWAL' that use the AsyncFSOutput in HDFS.
> >
> > >
> > > On Thu, Apr 28, 2016 at 9:42 PM, Duo Zhang 
> wrote:
> > > > Six month after I filed HBASE-14790...
> > > >
> > > > Now the AsyncFSWAL is ready. The WALPE result shows that it is
> > > *1.4x~3.7x*
> > > > faster than FSHLog. The ITBLL result turns out that it is *not bad*
> > than
> > > > FSHLog(the master branch is not that stable itself...).
> > > >
> > > > More details can be found on HBASE-15536.
> > > >
> > > > So here we propose to change the default WAL from FSHLog to
> AsyncFSWAL.
> > > > Suggestions are welcomed.
> > > >
> > > > Thanks.
> > >
> > >
> > >
> > > --
> > > busbey
> > >
> >
>

Re: [DISCUSS] Make AsyncFSWAL the default WAL in 2.0

[张铎]

2016-04-29 11:35 GMT+08:00 Ted Yu :

> bq. AsyncFSOutput will be in HDFS-3.0
>
> Is there HDFS JIRA for the above ? Can you share the number ?
>
I have not filed a new one but there are bunch of related issues already,
such as this one https://issues.apache.org/jira/browse/HDFS-916

>
> bq. Just wrap FSDataOutputStream to make it act like an asynchronous output
>
> Can you be a bit more specific ?
> HBase currently works with WASB and Azure Data Lake. Does the above mean
> their performance would suffer ?
>
Yes, the performance will suffer...
The fallback implementation is not aim to get a good performance, just for
compatibility with any FileSystem implementation.

>
> On Thu, Apr 28, 2016 at 8:30 PM, 张铎  wrote:
>
> > Inline comments.
> > Thanks,
> >
> > 2016-04-29 10:57 GMT+08:00 Sean Busbey :
> >
> > > I am nervous about having default out-of-the-box new HBase users
> reliant
> > on
> > > a bespoke HDFS client, especially given Hadoop's compatibility
> > > promises and history. Answers for these questions would make me more
> > > confident:
> > >
> > > 1) Where are we on getting the client-side changes to HDFS pushed back
> > > upstream?
> > >
> > No progress yet... Here I want to tell a good story that HBase is already
> > use it as default :)
> >
> > >
> > > 2) How well do we detect when our FS is not HDFS and what does
> > > fallback look like?
> > >
> > Just wrap FSDataOutputStream to make it act like an asynchronous
> > output(call hflush in a separated thread). The performance is not good I
> > think.
> >
> > >
> > > 3) Will this mean altering the versions of Hadoop we label as
> > > supported for HBase 2.y+?
> > >
> > I have tested with hadoop versions from 2.4.x to 2.7.x, so I don't think
> we
> > need to change the supported versions?
> >
> > >
> > > 4) How are we going to ensure our client remains compatible with newer
> > > Hadoop releases?
> > >
> > We can not ensure, HDFS always breaks HBase at a new release...
> > I need to test AsyncFSWAL on every new 2.x release and make it compatible
> > with that version. And back to #1, I think we should make sure that the
> > AsyncFSOutput will be in HDFS-3.0. And in HBase-3.0, we can introduce a
> new
> > 'AsyncFSWAL' that use the AsyncFSOutput in HDFS.
> >
> > >
> > > On Thu, Apr 28, 2016 at 9:42 PM, Duo Zhang 
> wrote:
> > > > Six month after I filed HBASE-14790...
> > > >
> > > > Now the AsyncFSWAL is ready. The WALPE result shows that it is
> > > *1.4x~3.7x*
> > > > faster than FSHLog. The ITBLL result turns out that it is *not bad*
> > than
> > > > FSHLog(the master branch is not that stable itself...).
> > > >
> > > > More details can be found on HBASE-15536.
> > > >
> > > > So here we propose to change the default WAL from FSHLog to
> AsyncFSWAL.
> > > > Suggestions are welcomed.
> > > >
> > > > Thanks.
> > >
> > >
> > >
> > > --
> > > busbey
> > >
> >
>

Re: [DISCUSS] Make AsyncFSWAL the default WAL in 2.0

[张铎]

Inline comments.
Thanks,

2016-04-29 10:57 GMT+08:00 Sean Busbey :

> I am nervous about having default out-of-the-box new HBase users reliant on
> a bespoke HDFS client, especially given Hadoop's compatibility
> promises and history. Answers for these questions would make me more
> confident:
>
> 1) Where are we on getting the client-side changes to HDFS pushed back
> upstream?
>
No progress yet... Here I want to tell a good story that HBase is already
use it as default :)

>
> 2) How well do we detect when our FS is not HDFS and what does
> fallback look like?
>
Just wrap FSDataOutputStream to make it act like an asynchronous
output(call hflush in a separated thread). The performance is not good I
think.

>
> 3) Will this mean altering the versions of Hadoop we label as
> supported for HBase 2.y+?
>
I have tested with hadoop versions from 2.4.x to 2.7.x, so I don't think we
need to change the supported versions?

>
> 4) How are we going to ensure our client remains compatible with newer
> Hadoop releases?
>
We can not ensure, HDFS always breaks HBase at a new release...
I need to test AsyncFSWAL on every new 2.x release and make it compatible
with that version. And back to #1, I think we should make sure that the
AsyncFSOutput will be in HDFS-3.0. And in HBase-3.0, we can introduce a new
'AsyncFSWAL' that use the AsyncFSOutput in HDFS.

>
> On Thu, Apr 28, 2016 at 9:42 PM, Duo Zhang  wrote:
> > Six month after I filed HBASE-14790...
> >
> > Now the AsyncFSWAL is ready. The WALPE result shows that it is
> *1.4x~3.7x*
> > faster than FSHLog. The ITBLL result turns out that it is *not bad* than
> > FSHLog(the master branch is not that stable itself...).
> >
> > More details can be found on HBASE-15536.
> >
> > So here we propose to change the default WAL from FSHLog to AsyncFSWAL.
> > Suggestions are welcomed.
> >
> > Thanks.
>
>
>
> --
> busbey
>

Re: [DISCUSS] Shade protobuf so we can move to a newer version

[张铎]

I think we could do it with some maven tricks. Just do not relocate the
protobuf things in asyncwal? Move the related classes to a new sub project?

Thanks.

2016-04-13 12:26 GMT+08:00 Sean Busbey :

> On Tue, Apr 12, 2016 at 6:17 PM, Stack  wrote:
> >
> >
> > On an initial pass, the only difficult part seems to be interaction with
> > HDFS in asyncwal (might just pull in the HDFS messages).
> >
> >
>
> I have some idea how we can make this work either by pushing asyncwal
> upstream to HDFS or through some maven tricks, depending on how much
> time we have.
>

Re: Please welcome new HBase committer Ashish Singhi

[张铎]

Congratulation!

2016-04-08 14:08 GMT+08:00 Heng Chen :

> congrats!
>
> 2016-04-08 13:26 GMT+08:00 Ted Yu :
>
> > Congrats, Ashish.
> >
> > > On Apr 7, 2016, at 10:23 PM, Stack  wrote:
> > >
> > > Ashish has contributed loads of stuff starting out basic doing rakes of
> > > simple fixes but then ramping up to take on the hard stuff going out of
> > his
> > > way to land the 'best' fix. He's been doing great work. Keep it up
> > Ashish!
> > >
> > > St.Ack
> >
>

Re: Please welcome new HBase Committer Francis Liu

[张铎]

Congratulation!

2016-04-08 14:09 GMT+08:00 Heng Chen :

> congrats!
>
>
> 2016-04-08 13:30 GMT+08:00 Jesse Yates :
>
> > Congrats and welcome!
> >
> > On Thu, Apr 7, 2016 at 10:27 PM Ted Yu  wrote:
> >
> > > Congratulations, Francis.
> > >
> > > > On Apr 7, 2016, at 10:19 PM, Stack  wrote:
> > > >
> > > > Francis has been around forever looking after one of the biggest
> HBase
> > > > deploys. He has contributed a bunch of big features during this time
> --
> > > > namespacing and grouping to mention a few -- and has more coming down
> > the
> > > > pipe.
> > > >
> > > > Please welcome Francis to the committer fold.
> > > >
> > > > Thanks for all the great work Francis,
> > > > St.Ack
> > >
> >
>

Re: [DISCUSS] No regions on Master node in 2.0

[张铎]

Agree on the performance concerns. IMO we should not hurt the performance
of small(maybe normal?) clusters when scaling for huge clusters.
And I also agree that the current implementation which allows Master to
carry system regions is not good(sorry for the chinglish...). At least, it
makes the master startup really complicated.

So IMO, we should let the master process or master machine to also carry
system regions, but in another way. Start another RS instance on the same
machine or in the same JVM? Or build a new storage based on the procedure
store and convert it to a normal table when it is too large?

Thanks.

2016-04-08 16:42 GMT+08:00 Elliott Clark :

> # Without meta on master, we double assign and lose data.
>
> That is currently a fact that I have seen over and over on multiple loaded
> clusters. Some abstract clean up of deployment vs losing data is a
> no-brainer for me. Master assignment, region split, region merge are all
> risky, and all places that HBase can lose data. Meta being hosted on the
> master makes communication easier and less flakey. Running ITBLL on a loop
> that creates a new table every time, and without meta on master everything
> will fail pretty reliably in ~2 days. With meta on master things pass MUCH
> more.
>
> # Master hosting the system tables locates the system tables as close as
> possible to the machine that will be mutating the data.
>
> Data locality is something that we all work for. Short circuit local reads,
> Caching blocks in jvm, etc. Bringing data closer to the interested party
> has a long history of making things faster and better. Master is in charge
> of just about all mutations of all systems tables. It's in charge of
> changing meta, changing acls, creating new namespaces, etc. So put the
> memstore as close as possible to the system that's going to mutate meta.
>
> # If you want to make meta faster then moving it to other regionservers
> makes things worse.
>
> Meta can get pretty hot. Putting it with other regions that clients will be
> trying to access makes everything worse. It means that meta is competing
> with user requests. If meta gets served and other requests don't, causing
> more requests to meta; or requests to user regions get served and other
> clients get starved.
> At FB we've seen read throughput to meta doubled or more by swapping it to
> master. Writes to meta are also much faster since there's no rpc hop, no
> queueing, to fighting with reads. So far it has been the single biggest
> thing to make meta faster.
>
>
> On Thu, Apr 7, 2016 at 10:11 PM, Stack  wrote:
>
> > I would like to start a discussion on whether Master should be carrying
> > regions or not. No hurry. I see this thread going on a while and what
> with
> > 2.0 being a ways out yet, there is no need to rush to a decision.
> >
> > First, some background.
> >
> > Currently in the master branch, HMaster hosts 'system tables': e.g.
> > hbase:meta. HMaster is doing more than just gardening the cluster,
> > bootstrapping and keeping all up and serving healthy as in branch-1; in
> > master branch, it is actually in the write path for the most critical
> > system regions.
> >
> > Master is this way because HMaster and HRegionServer servers have so much
> > in common, they should be just one binary, w/ HMaster as any other server
> > with the HMaster function a minor appendage runnable by any running
> > HRegionServer.
> >
> > I like this idea, but the unification work was just never finished. What
> is
> > in master branch is a compromise. HMaster is not a RegionServer but a
> > sort-of RegionServer doing part serving. So we have HMaster role, a new
> > part-RegionServer-carrying-special-regions role and then a full-on
> > HRegionServer role. We need to fix this messyness. We could revert to
> plain
> > branch-1 roles or carrying the
> > HMaster-function-is-something-any-RegionServer-could-execute through to
> > completion.
> >
> > More background from a time long-past with good comments by the likes of
> > our Francis Liu and Mighty Matteo Bertozzi are here [1], on unifying
> master
> > and meta-serving. Slightly related are old discussions on being able to
> > scale by splitting meta with good comments by our Elliott Clark [2].
> >
> > Also for consideration, the landscape has since changed. [1] was written
> > before we had ProcedureV2 available to us where we could record
> > intermediate transition states local to the Master rather than remote as
> > intermediate updates to an hbase:meta over rpc running on another node.
> >
> > Enough on the background.
> >
> > Let me provoke discussion by making the statement that we should undo
> > HMaster carrying any regions ever; that the HMaster function is work
> enough
> > for a single dedicated server and that it important enough that it cannot
> > take a background role on a serving RegionServer (I could go back from
> this
> > position if evidence HMaster role could be backgrounded). Notions of a
> > Master carrying system

Re: [DISCUSS] hbase-2.0.x requires jdk8

[张铎]

+1 on jdk8 only. Bunch of code could be refactored with jdk8's new
features, new classes and new methods.
For example, most 'get->putIfAbsent->checkAndCleanUp' and 'get->lock->get
again->put->unlock' can be replaced by computeIfAbsent.

2016-04-08 16:43 GMT+08:00 Elliott Clark :

> As a side benefit, only testing one jdk would seriously speed up test runs.
>
> +1
>
> On Thu, Apr 7, 2016 at 10:12 PM, Stack  wrote:
>
> > Is there a YETUS issue for this Sean or should I file one?
> > St.Ack
> >
> > On Thu, Apr 7, 2016 at 9:21 PM, Sean Busbey  wrote:
> >
> > > we'll need to get a feature into yetus that can change multijdk
> > > settings by branch, presuming that change will result in us either
> > > taking on jdk8-only dependencies or direct use of jdk8-only features.
> > >
> > > On Thu, Apr 7, 2016 at 9:18 PM, Stack  wrote:
> > > > Any objection?
> > > >
> > > > jdk7 is dead, EOL'd.
> > > >
> > > > You all good w/ this?
> > > >
> > > > St.Ack
> > >
> > >
> > >
> > > --
> > > busbey
> > >
> >
>

Re: Branch for 1.3

[张铎]

At least include HBASE-15400 in 1.3 please...
Otherwise these things can only be integrated in 1.4 since DTCP can not be
used together with DefaultStoreEngine after HBASE-15400 getting in.

2016-03-22 2:03 GMT+08:00 Enis Söztutar :

> You may want to track https://issues.apache.org/jira/browse/HBASE-15339 as
> a parent for date-tiered compaction improvements. Current compaction policy
> is useful in its own, but handling existing data, bulk loading etc will be
> improved with these subtasks. I think the patches can land before the 1.3
> timeframe, but of course open for discussion for inclusion. My vote would
> be to include all the improvements since it would be easier to tell the
> story to users.
>
> Enis
>
> On Mon, Mar 21, 2016 at 3:32 AM, Ted Yu  wrote:
>
> > For Spark connector, we should start a separate discussion thread about
> > backporting to branch-1.
> >
> > Zhan has a bug fix coming this week which deals with how negative numbers
> > are handled in comparison.
> >
> > FYI
> >
> > On Mon, Mar 21, 2016 at 1:35 AM, Mikhail Antonov 
> > wrote:
> >
> > > Hi folks,
> > >
> > > bringing this topic up again. I'm planning to start spinning 1.3 builds
> > and
> > > see if/where they break in a week or two, and (depending on how it
> does)
> > > start preparing RCs in a month or maybe two. So, let's see where we
> are.
> > >
> > > Big items first. There were long debates around three big items - MOBs,
> > > Spark connector and RS groups, whether we should have them or not.
> > >
> > >  - MOBs
> > > I believe we decided that they aren't going to go in branch-1, and
> hence
> > > not in branch-1.3 for sure. We might get back to that debate and
> > > re-consider MOBs for branch-1 if 2.0 is delayed, but almost certainly
> > they
> > > won't make it in 1.3 timeframe anyway as I feel.
> > >
> > > - Spark connector - HBASE-14160 it looks like it has 3 subtasks open,
> one
> > > of which is big one (HBASE-14375) - define public API for Spark
> > > integration. From the Jira looks like active work is happening on other
> > > subtasks, but not on this one. So how's public API going? How stable it
> > is?
> > > Who would want to have Spark in 1.3 and willing to help with this?
> OTOH -
> > > who has objections about back-porting it? Has anyone been using it in
> > some
> > > real environment?
> > >
> > >  - RS groups - there was recently a thread about them, I'd like to
> bring
> > it
> > > up again and get to some conclusion.
> > >
> > > Other features which we had in flight a month ago -
> > >
> > >  - HBASE-15181 - date based tiered compactions has landed
> > >  - HBASE-11290 - unlock RegionStates. I'm afraid the codebase has moved
> > > forward quite a bit since the benchmark was run on this change :( -
> > Francis
> > > - are you using it now? If we could have some benchmarks on the latest
> > > rebase that I think would be great.
> > >
> > >
> > >  - HBASE-13557 - special handling for system tables WALs - should we
> > still
> > > keep it targeted for 1.3?
> > >  - HBASE-13017 - keep table state in meta, this one doesn't look like
> > it's
> > > going to make it in
> > >
> > > As a new item on my list - I'm looking forward to see more of
> HBASE-15492
> > > (memory optimizations) subtasks to go in branch-1.
> > >
> > > Thanks!
> > > Mikhail
> > >
> > > On Sun, Feb 28, 2016 at 10:53 AM, Andrew Purtell <
> > andrew.purt...@gmail.com
> > > >
> > > wrote:
> > >
> > > > I'm starting a push at work to get us up on 1.2. Assuming that
> happens
> > > > later this year I think that will be the end of my close attention to
> > > 0.98.
> > > >
> > > > > On Feb 26, 2016, at 1:54 PM, Nick Dimiduk 
> > wrote:
> > > > >
> > > > > On Fri, Feb 26, 2016 at 12:23 PM, Andrew Purtell <
> > apurt...@apache.org>
> > > > > wrote:
> > > > >
> > > > >> In the meantime those of us running HBase in production would
> > benefit
> > > > from
> > > > >> fairly frequent minor releases.
> > > > >
> > > > > +1. Having to look back to 0.98 to get some new feature is
> > problematic.
> > > > >
> > > > >> On Fri, Feb 26, 2016 at 9:50 AM, Elliott Clark  >
> > > > wrote:
> > > > >>
> > > > >>> I disagree. We have agreed that 2.0 will have a new assignement
> > > > manager.
> > > > >>> There's a lot of work that has been done on getting that in, so
> far
> > > > there
> > > > >>> are no benefits to the end user from all that work. We should
> stick
> > > > with
> > > > >>> the plan and release 2.0 when it's ready.
> > > > >>>
> > > > >>> On Fri, Feb 26, 2016 at 9:39 AM, Stephen Jiang <
> > > > syuanjiang...@gmail.com>
> > > > >>> wrote:
> > > > >>>
> > > >  Thanks for Mikhail for taking the 1.3 RM role.  Looks like we
> > have a
> > > > >> lot
> > > > >>> of
> > > >  new things in 1.3 release.
> > > > 
> > > >  Based on the experience of 1.1 and 1.2 release, it takes a lot
> of
> > > > >> efforts
> > > >  to get a stable minor release out.  From this, I have my own
> > 2-cents
> > > > on
> > > > >>> 1.4
> > > >  release.  

Re: [ANNOUNCE] New HBase committer Yu Li

[张铎]

Congratulations!

2016-03-17 10:03 GMT+08:00 Ted Yu :

>  Congratulations, Yu.
>
> Keep up the good work.
>
> On Wed, Mar 16, 2016 at 6:49 PM, Nick Dimiduk  wrote:
>
> > On behalf of the Apache HBase PMC, I am pleased to announce that Yu Li
> > has accepted the PMC's invitation to become a committer on the
> > project. We appreciate all of Yu's generous contributions thus far and
> > look forward to his continued involvement.
> >
> > Congratulations and welcome, Yu!
> >
> > -n
> >
>

Re: [VOTE] HBase 1.2.0 RC2

[张铎]

HBASE-15252 is fixed :).

2016-02-12 14:00 GMT+08:00 Stack :

> -1
>
> The dataloss issue was just discovered. I think now we know of it, even
> though the incidence is rare, would be best to respin the RC.
>
> You the man Sean,
> St.Ack
>
>
> On Thu, Feb 11, 2016 at 8:59 PM, Stack  wrote:
>
> > On Thu, Feb 11, 2016 at 5:04 PM, Sean Busbey 
> > wrote:
> >
> >> On Feb 11, 2016 18:33, "张铎"  wrote:
> >> >
> >> > Should we include HBASE-15252? It is a data loss issue.
> >> >
> >>
> >> It's marked major (though perhaps that's off since it's dataloss, even
> if
> >> rare). More importantly it's been present in prior releases for some
> time.
> >>
> >> Blocking 1.2.0 would put pressure on getting a solution faster, I think.
> >> Additionally, letting the fix wait for 1.2.1 will give me a good
> incentive
> >> to keep the path releases on schedule. ;)
> >>
> >> My 2¢. Happy to roll another RC if folks see it otherwise.
> >>
> >
> > Dataloss. I think we should roll a new RC with short voting timeframe.
> > St.Ack
> >
> >
>

Re: [VOTE] HBase 1.2.0 RC2

[张铎]

Should we include HBASE-15252? It is a data loss issue.

2016-02-12 7:34 GMT+08:00 Elliott Clark :

> +1
>
> Been running something close to RC in production for a while now.
> Checked the layout everything looks good
> Running ITBLL in a loop and there's no data loss. ( Sometimes the master
> still gets stuck but that's not new ).
>
>
> On Thu, Feb 11, 2016 at 3:22 PM, Stack  wrote:
>
> > +1
> >
> > Checked signature and hash.
> >
> > Reviewed layout.
> >
> > Ran rat check.
> >
> > Built src and site.
> >
> > Doc looks good.
> >
> > Started it all up and loaded and verified data. Nothing odd in the logs.
> >
> > I've been running RCs on a cluster doing ITBLL with monkeys over last few
> > weeks. 1.2.0 stays up and running in my runs (9-node, 1B nodes, done a
> few
> > times).
> >
> > St.Ack
> >
> >
> > On Mon, Feb 8, 2016 at 7:46 AM, Sean Busbey  wrote:
> >
> > > Hi Folks!
> > >
> > > I'm pleased to announce the second release candidate for HBase 1.2.0.
> > >
> > > Artifacts are available here:
> > >
> > > https://dist.apache.org/repos/dist/dev/hbase/hbase-1.2.0RC2/
> > >
> > > As of this vote, the relevant md5 hashes are:
> > >
> > > 836f8401462b36e021a8b6bd55aff4ec  hbase-1.2.0-bin.tar.gz
> > > 9b1d592a17a4c167043cb0ca8d8f308b  hbase-1.2.0-src.tar.gz
> > >
> > > Maven artifacts are available in this staging repository:
> > >
> > >
> https://repository.apache.org/content/repositories/orgapachehbase-1128/
> > >
> > > All artifacts are signed with my code signing key 0D80DB7C, available
> in
> > > the project KEYS file:
> > >
> > > http://www.apache.org/dist/hbase/KEYS
> > >
> > > these artifacts correspond to commit hash
> > >
> > > d568db8372a3fbc6b93c5854749c30276ba19ba4
> > >
> > > which signed tag 1.2.0RC2 currently point to
> > >
> > >
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=hbase.git;a=tag;h=6a689261c76d1679c6a71bd3a0cbd87a185bff27
> > >
> > > HBase 1.2.0 is the second minor release in the HBase 1.x line,
> continuing
> > > on
> > > the theme of bringing a stable, reliable database to the Hadoop and
> NoSQL
> > > communities. This release includes roughly 250 resolved issues not
> > covered
> > > by previous 1.x releases.
> > >
> > > Notable new features include:
> > > - JDK8 is now supported
> > > - Hadoop 2.6.1+ and Hadoop 2.7.1+ are now supported
> > > - per column-family time ranges for scan (HBASE-14355)
> > > - daemons respond to SIGHUP to reload configs (HBASE-14529)
> > > - region location methods added to thrift2 proxy (HBASE-13698)
> > > - table-level sync that sends deltas (HBASE-13639)
> > > - client side metrics via JMX (HBASE-12911)
> > >
> > > The full list of issues can be found at:
> > >
> > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12332062
> > >
> > > To see the changes since the prior release candidate, you can use the
> > > following git command on
> > > a up-to-date checkout of the hbase repository:
> > >
> > > git log 1.2.0RC1..1.2.0RC2
> > >
> > > Please take a few minutes to verify the release[1] and vote on
> releasing
> > > it:
> > >
> > > [ ] +1 Release this package as Apache HBase 1.2.0
> > > [ ] +0 no opinion
> > > [ ] -1 Do not release this package because...
> > >
> > > Vote will be subject to Majority Approval[2] and will close at 4:00PM
> UTC
> > > on Monday, Feb 15th, 2016[3].
> > >
> > > [1]: http://www.apache.org/info/verification.html
> > > [2]: https://www.apache.org/foundation/glossary.html#MajorityApproval
> > > [3]: to find this in your local timezone see:
> > > http://s.apache.org/hbase-1.2.0-rc2-close
> > >
> >
>

Re: [DISCUSS] Dynamic switches in master

[张铎]

My opinion, use conf file as default, and use code to override the config
in conf file.
And we need to persist the overridden value somewhere, not zk I'd say. I
agree with Matteo that we should reduce the dependency of zk. Maybe we
could introduce an hbase:config table?

And for the dynamic configuration framework, it is difficult to treat all
configurations in the same way I think. I prefer changing all
configurations with a shell command(the actual implementation should an RPC
call to master maybe), but if we have regionservers with different
hardwares in cluster, they may have different configurations so modify conf
file directly on the machine is a more proper way? I have no idea...

Thanks.

2016-02-04 11:40 GMT+08:00 Enis Söztutar :

> This came up in the review for HBASE-15128, but we thought that we should
> surface this discussion in dev@.
>
> The idea for HBASE-15128 is that, we should have a dynamic switch in master
> so that the operator (or HBCK) can disable all region splits or merges.
>
> We currently have such dynamic switches for balancer and normalizer which
> keep their state in zk. We also have a switch for catalog janitor which
> keeps its state only in memory (not sure why). For the new switches Heng
> Chen implemented a "set_switch" RPC to master, a switch type, a zk node.
> The idea is to move balancer, catalog janitor and normalizer switches to
> this after this patch.
>
> However, Matteo raised a point about this framework being an alternative
> way of doing dynamic configuration. The problem is that we do not have a
> dynamic conf framework that we can piggyback on, and since balancer and
> other switches already work this way, it does not make sense to block the
> issue. What do you guys think? If we want to implement dynamic conf, how
> would we handle things like enabling / disabling balancer from conf and or
> from code? Any good ideas?
>
> Enis
>

Re: DISCUSS: Protobufs?

[张铎]

Since they already implement an unsafeWrap method which breaks immutability
so add DBB support for CIS does not break anything else. I think it is easy
to make the google guys accept the PR. Let's do it :)

2016-02-03 5:58 GMT+08:00 Stack :

> Looks like the boys from pb are doing COS only, not CIS, and suggest pull
> request. I'll have a go unless someone else wants to.
> St.Ack
>
> On Tue, Feb 2, 2016 at 11:41 AM, Enis Söztutar  wrote:
>
> > Google guys over at
> > https://github.com/grpc/grpc-java/issues/1054#issuecomment-147295224 are
> > saying that CIS changes may be coming to 2.x from what I understand. If
> so,
> > our life would be easier. Even so, I'm 100% sure we have to do shading
> > since Hadoop will not change it's PB dependency anytime soon.
> >
> > We have to do this before doing shading:
> > https://issues.apache.org/jira/browse/HBASE-15174
> >
> > Enis
> >
> > On Tue, Feb 2, 2016 at 8:15 AM, Stack  wrote:
> >
> > > Thanks Duo. If proto3 had what we wanted, you are suggesting we might
> > move
> > > to proto3 setting it to do proto2 support and shade it so we don't
> clash
> > > with other includes of pb?
> > >
> > > Regards Anoop comment, the note on the end of this issue looks
> promising
> > > but I don't know when it'd see the light of day:
> > > https://github.com/grpc/grpc-java/issues/1054#issuecomment-147295224
> > >
> > > St.Ack
> > >
> > >
> > > On Mon, Feb 1, 2016 at 10:49 PM, Anoop John 
> > wrote:
> > >
> > > > UnsafeByteStrings - This may help us to avoid copy even with out our
> > > > HBaseZeroCopyByteString stuff.  But with a DirectByteBuffer, it has
> to
> > > copy
> > > > data to onheap byte[].   We even want a DBB backing !
> > > >
> > > > -Anoop-
> > > >
> > > > On Tue, Feb 2, 2016 at 12:07 PM, 张铎  wrote:
> > > >
> > > > > https://groups.google.com/forum/#!topic/protobuf/wAqvtPLBsE8
> > > > >
> > > > > PB2 and PB3 are wire compatible, and of course, protobuf-java is
> not
> > > > > compatible so dependency will be a problem... But I think the
> shaded
> > > > client
> > > > > and server can solve the problem?
> > > > >
> > > > > Thanks.
> > > > >
> > > > > 2016-02-02 14:27 GMT+08:00 Stack :
> > > > >
> > > > > > We are running into a few issues with protobufs.
> > > > > >
> > > > > > + PB always copies all data before making a Message. This
> generates
> > > > > garbage
> > > > > > unnecessarily.
> > > > > > + CodedInputStream does not support ByteBuffers in 2.5. In 2.6 it
> > > does
> > > > > but
> > > > > > again, copies the data out of the BB always; this is especially
> > > painful
> > > > > > when the BB is a DBB with its data offheap and intent is to keep
> > data
> > > > > > offheap.
> > > > > >
> > > > > > There are other issues. CIS allocates 4k buffers regardless (See
> > > > > > HBASE-15177).
> > > > > > And then there was the HBaseZeroCopyByteString fun and games we
> > had a
> > > > > while
> > > > > > back.
> > > > > >
> > > > > > 3.0 PB adds UnsafeByteStrings so can do zero copy. Thats good.
> But
> > > PB3
> > > > is
> > > > > > incompatible with PB2 (or at least, it looks like PB2 clients
> can't
> > > > talk
> > > > > to
> > > > > > PB3 [1]).
> > > > > >
> > > > > > There is javanano protobufs. All is open access, but it too looks
> > > > > different
> > > > > > to PB2 (i've not tried it).
> > > > > >
> > > > > > Protostuff seems really quiet these times [2].
> > > > > >
> > > > > > Fork (and shade)?
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > St.Ack
> > > > > >
> > > > > > 1. https://github.com/google/protobuf/releases
> > > > > > 2. https://groups.google.com/forum/#!forum/protostuff
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: DISCUSS: Protobufs?

[张铎]

https://groups.google.com/forum/#!topic/protobuf/wAqvtPLBsE8

PB2 and PB3 are wire compatible, and of course, protobuf-java is not
compatible so dependency will be a problem... But I think the shaded client
and server can solve the problem?

Thanks.

2016-02-02 14:27 GMT+08:00 Stack :

> We are running into a few issues with protobufs.
>
> + PB always copies all data before making a Message. This generates garbage
> unnecessarily.
> + CodedInputStream does not support ByteBuffers in 2.5. In 2.6 it does but
> again, copies the data out of the BB always; this is especially painful
> when the BB is a DBB with its data offheap and intent is to keep data
> offheap.
>
> There are other issues. CIS allocates 4k buffers regardless (See
> HBASE-15177).
> And then there was the HBaseZeroCopyByteString fun and games we had a while
> back.
>
> 3.0 PB adds UnsafeByteStrings so can do zero copy. Thats good. But PB3 is
> incompatible with PB2 (or at least, it looks like PB2 clients can't talk to
> PB3 [1]).
>
> There is javanano protobufs. All is open access, but it too looks different
> to PB2 (i've not tried it).
>
> Protostuff seems really quiet these times [2].
>
> Fork (and shade)?
>
> Thoughts?
>
> St.Ack
>
> 1. https://github.com/google/protobuf/releases
> 2. https://groups.google.com/forum/#!forum/protostuff
>

Re: Please welcome new HBase committer Stephen Yuan Jiang

[张铎]

Congratulations!

2015-08-21 10:12 GMT+08:00 Ted Yu :

> Congratulations Stephen.
>
>
>
> > On Aug 20, 2015, at 7:09 PM, Andrew Purtell  wrote:
> >
> > On behalf of the Apache HBase PMC, I am pleased to announce that Stephen
> > Jiang has accepted the PMC's invitation to become a committer on the
> > project. We appreciate all of Stephen's hard work and generous
> > contributions thus far, and look forward to his continued involvement.
> >
> > Congratulations and welcome, Stephen!
> >
> > --
> > Best regards,
> >
> >   - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
>

Re: [ANNOUNCE] New HBase committer Mikhail Antonov

[张铎]

Congratulations!

2015-07-01 8:50 GMT+08:00 Srikanth Srungarapu :

> Kudos, Mikhail!
>
> On Tue, Jun 30, 2015 at 5:42 PM Sean Busbey  wrote:
>
> > congrats Mikhail!
> >
> > On Tue, Jun 30, 2015 at 7:14 PM, Andrew Purtell 
> > wrote:
> >
> > > On behalf of the Apache HBase PMC, I am pleased to announce that
> Mikhail
> > > Antonov has accepted the PMC's invitation to become a committer on the
> > > project. We appreciate all of Mikhail's generous contributions thus far
> > and
> > > look forward to his continued involvement.
> > >
> > > Congratulations and welcome, Mikhail!
> > >
> > > --
> > > Best regards,
> > >
> > >- Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> > >
> >
> >
> >
> > --
> > Sean
> >
> --
> Thanks,
> Srikanth.
>

Re: [ANNOUNCE] New HBase committer Esteban Gutierrez

[张铎]

Congratulations!

2015-07-01 8:40 GMT+08:00 Jerry He :

> Congrats and Welcome, Esteban!
>
> On Tue, Jun 30, 2015 at 5:37 PM, Apekshit Sharma 
> wrote:
>
> > Congrats Esteban!
> >
> > On Tue, Jun 30, 2015 at 5:33 PM, Jesse Yates 
> > wrote:
> >
> > > Congrats and welcome!
> > >
> > > On Tue, Jun 30, 2015, 5:33 PM Sean Busbey  wrote:
> > >
> > > > congrats Esteban!
> > > >
> > > > On Tue, Jun 30, 2015 at 7:15 PM, Andrew Purtell  >
> > > > wrote:
> > > >
> > > > > On behalf of the Apache HBase PMC, I am pleased to announce that
> > > Esteban
> > > > > Gutierrez has accepted the PMC's invitation to become a committer
> on
> > > the
> > > > > project. We appreciate all of Esteban's generous contributions thus
> > far
> > > > and
> > > > > look forward to his continued involvement.
> > > > >
> > > > > Congratulations and welcome, Esteban!
> > > > >
> > > > > --
> > > > > Best regards,
> > > > >
> > > > >- Andy
> > > > >
> > > > > Problems worthy of attack prove their worth by hitting back. - Piet
> > > Hein
> > > > > (via Tom White)
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Sean
> > > >
> > >
> >
> >
> >
> > --
> >
> > Regards
> >
> > Apekshit Sharma | Software Engineer, Cloudera | Palo Alto, California |
> > 650-963-6311
> >
>

Re: Github PRs

[张铎]

How does spark make use of Github? Seems they only use PRs to accept
patches but still use apache jira.

And for now, set up a bot that automatically comments on new PR to tell the
contributor open issue on apache jira?

2015-06-25 3:45 GMT+08:00 Sean Busbey :

> So for now comments on each pointing folks to the contributor guide and
> jira would work, right?
>
> On Wed, Jun 24, 2015 at 2:09 PM, Andrew Purtell 
> wrote:
>
> > I don't think we should accept PRs until we've discussed it and there is
> > consensus to do it. As you can see we are not set up with the necessary
> > integration to support that.
> >
> > Not sure how we can document better we don't currently care about our
> > GitHub mirrors, but that would be good.
> >
> >
> > > On Jun 24, 2015, at 11:52 AM, Sean Busbey  wrote:
> > >
> > > Hey folks,
> > >
> > > Just noticed we have a number of outstanding pull requests on the
> github
> > > mirror.
> > >
> > > https://github.com/apache/hbase/pulls
> > >
> > > Anyone know if we're set to get emails on dev@ when these open? Can we
> > try
> > > to get them taken care of, even if that just means pointing folks to
> > JIRA?
> > >
> > > --
> > > Sean
> >
>
>
>
> --
> Sean
>

Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.

[张铎]

Is there any comparison between HBASE-11339 and HDFS-7240?
Is their 'Object' a super set of our 'Medium Object'?

2015-05-21 10:38 GMT+08:00 Ted Yu :

> This is a useful feature, Jon.
>
> I went over the mega-patch and left some comments on review board.
>
> I noticed that hbck was not included in the patch. Neither did I find a
> sub-task of HBASE-11339 that covers hbck.
>
> Do you or Jingcheng plan to add MOB-aware capability for hbck ?
>
> Cheers
>
> On Wed, May 20, 2015 at 9:21 AM, Jonathan Hsieh  wrote:
>
> > Hi folks,
> >
> > The Medium Object (MOB) Storage feature (HBASE-11339[1]) is modified I/O
> > and compaction path that allows individual moderately sized values
> > (10k-10MB) to be stored so that write amplification is reduced when
> > compared to the normal I/O path.   At a high level, it provides alternate
> > flush and compaction mechanisms that segregates large cells into a
> separate
> > area where they are not subject to potentially frequent compaction and
> > splits that can be encountered in the normal I/O path. A more detailed
> > design doc can be found on the hbase-11339 jira.
> >
> > Jingcheng Du has been working on the mob feature for a while and Anoop,
> Ram
> > and I have been shepherding him through the design revisions and
> > implementation of the feature in the hbase-11339 branch.[2]
> >
> > The branch we are proposing to merge into master is compatible with
> HBase's
> > core functionality including snapshots, replication, shell support,
> behaves
> > well with table alters, bulk loads and does not require external MR
> > processes. It has been documented, and subject to many integration test
> > runs  (ITBLL, ITAcidGuarantees, ITIngest) including fault injection.
> > Performance testing of the feature shows what can be a 2x-3x throughput
> > improvement for workloads that contain mobs. These results can be seen on
> > the hbase 2.0 panel discussion slides from hbasecon (once published).
> >
> > Recently there have been some hfile encryption related shortcomings that
> we
> > could address in branch or in master.
> >
> > Earlier iterations of the feature has been tested in production by users
> > that Jingcheng has been responsible for.  A version has also been
> deployed
> > at users I have been responsible for.  Some of the folks from Huawei
> > (ashutosh) have also been submitting the recent encryption bug reports
> > against the hbase-11339 branch so there is some evidence of usage by
> them.
> >
> > The four of us  (Jingcheng, Ram, Anoop and I) are satisfied with the
> > feature and feel it is a good time to call a merge vote.  Ive posted a
> > megapatch version for folks who want to peruse the code. [3]
> >
> > What do you all think?
> >
> > Thanks,
> > Jingcheng, Jon, Ram, and Anoop.
> >
> > [1] https://issues.apache.org/jira/browse/HBASE-11339
> > [2] https://github.com/apache/hbase/tree/hbase-11339
> > [3] https://reviews.apache.org/r/34475/
> > --
> > // Jonathan Hsieh (shay)
> > // HBase Tech Lead, Software Engineer, Cloudera
> > // j...@cloudera.com // @jmhsieh
> >
>

Re: A compact bug?

[张铎]

These code is wrapped by a 'if (this.compaction == null)', so the code will
only be executed when selectNow == false.

2015-04-21 22:49 GMT+08:00 Ted Yu :

> In CompactionRunner#run(), we have:
>
> // Now see if we are in correct pool for the size; if not, go to
> the correct one.
>
> // We might end up waiting for a while, so cancel the selection.
>
> assert this.compaction.hasSelection();
>
> ThreadPoolExecutor pool = store.throttleCompaction(
>
> compaction.getRequest().getSize()) ? longCompactions :
> shortCompactions;
>
> if (this.parent != pool) {
>
>   this.store.cancelRequestedCompaction(this.compaction);
>
> The above code would adjust to the correct pool, right ?
>
>
> Cheers
>
> On Tue, Apr 21, 2015 at 7:10 AM, 张铎  wrote:
>
>> I think it is a bug. According to the comment above, they just want to
>> put system compactions(selectNow == false) to the small pool, so the code
>> should like this
>>
>> ThreadPoolExecutor pool;
>> if (selectNow) {
>>   pool = s.throttleCompaction(compaction.getRequest().getSize())
>> ? longCompactions : shortCompactions;
>> } else {
>>   pool = shortCompactions;
>> }
>>
>> My code is on master so ignore the variable name differences...
>>
>> Would you mind open a issue on jira?
>>
>> Thanks.
>>
>> 2015-04-21 21:10 GMT+08:00 zhou_shuaif...@sina.com <
>> zhou_shuaif...@sina.com>:
>>
>>> Hi all,
>>>
>>> In compactsplitthread.java, the requestCompactionInternal method is like
>>> this:
>>>
>>>   private synchronized CompactionRequest requestCompactionInternal(final
>>> HRegion r, final Store s,
>>> ...
>>>
>>> ThreadPoolExecutor pool = (!selectNow && s.throttleCompaction(size))
>>>   ? largeCompactions : smallCompactions;
>>>
>>> ...
>>>
>>> this will cause all compactions with selectNow =true go to the small
>>> compaction queue, even the size is large enough, is it reasonable?
>>>
>>> I checked the commit history, this comes from HBASE-8665, is there any
>>> reason or a bug?
>>>
>>> thanks.
>>>
>>>
>>> zhou_shuaif...@sina.com
>>>
>>
>>
>

Re: A compact bug?

[张铎]

I think it is a bug. According to the comment above, they just want to put
system compactions(selectNow == false) to the small pool, so the code
should like this

ThreadPoolExecutor pool;
if (selectNow) {
  pool = s.throttleCompaction(compaction.getRequest().getSize())
? longCompactions : shortCompactions;
} else {
  pool = shortCompactions;
}

My code is on master so ignore the variable name differences...

Would you mind open a issue on jira?

Thanks.

2015-04-21 21:10 GMT+08:00 zhou_shuaif...@sina.com 
:

> Hi all,
>
> In compactsplitthread.java, the requestCompactionInternal method is like
> this:
>
>   private synchronized CompactionRequest requestCompactionInternal(final
> HRegion r, final Store s,
> ...
>
> ThreadPoolExecutor pool = (!selectNow && s.throttleCompaction(size))
>   ? largeCompactions : smallCompactions;
>
> ...
>
> this will cause all compactions with selectNow =true go to the small
> compaction queue, even the size is large enough, is it reasonable?
>
> I checked the commit history, this comes from HBASE-8665, is there any
> reason or a bug?
>
> thanks.
>
>
> zhou_shuaif...@sina.com
>

Re: Please welcome new HBase committer Jing Chen (Jerry) He

[张铎]

Congratulations!

2015-04-02 15:56 GMT+08:00 Andrey Stepachev :

> Congrats, Jerry!
>
> On Thu, Apr 2, 2015 at 8:34 AM, abhishek kr 
> wrote:
>
> > Congrats, Jerry.
> >
> > Regards,
> > Abhishek
> >
> > -Original Message-
> > From: Ted Yu [mailto:yuzhih...@gmail.com]
> > Sent: 02 April 2015 01:56
> > To: u...@hbase.apache.org
> > Cc: dev@hbase.apache.org
> > Subject: Re: Please welcome new HBase committer Jing Chen (Jerry) He
> >
> > Congratulations, Jerry.
> >
> > On Wed, Apr 1, 2015 at 10:53 AM, Andrew Purtell 
> > wrote:
> >
> > > On behalf of the Apache HBase PMC, I am pleased to announce that Jerry
> > > He has accepted the PMC's invitation to become a committer on the
> > > project. We appreciate all of Jerry's hard work and generous
> > > contributions thus far, and look forward to his continued involvement.
> > >
> > > Congratulations and welcome, Jerry!
> > >
> > > --
> > > ​​
> > >
> > > Best regards,
> > >
> > >- Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> > > Hein (via Tom White)
> > >
> >
>
>
>
> --
> Andrey.
>

Re: Please welcome new HBase committer Srikanth Srungarapu

[张铎]

Congratulations!

2015-04-02 15:56 GMT+08:00 Andrey Stepachev :

> Congratulations, Srikanth!
>
> On Thu, Apr 2, 2015 at 7:57 AM, abhishek kr 
> wrote:
>
> > Congrats, Srikanth.
> >
> > Regards,
> > Abhishek
> >
> > -Original Message-
> > From: Andrew Purtell [mailto:apurt...@apache.org]
> > Sent: 02 April 2015 01:53
> > To: dev@hbase.apache.org; u...@hbase.apache.org
> > Subject: Please welcome new HBase committer Srikanth Srungarapu
> >
> > On behalf of the Apache HBase PMC, I am pleased to announce that Srikanth
> > Srungarapu has accepted the PMC's invitation to become a committer on the
> > project. We appreciate all of Srikanth's hard work and generous
> > contributions thus far, and look forward to his continued involvement.
> >
> > Congratulations and welcome, Srikanth!
> >
> > --
> > ​​
> >
> > Best regards,
> >
> >- Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
>
>
> --
> Andrey.
>

Re: Testing and CI -- Apache Jenkins Builds (WAS -> Re: Testing)

[张铎]

Make findbugs and checkstyle plugins always run.
The default behavior only publishes result for stable builds, but master is
not always stable and sometimes we introduce new warnings in unstable
builds(see builds 6310-6314).
And findbugs and checkstyle will not fail unless we abort the building
process I think, so it is safe to turn it on every time.

Thanks.

2015-03-22 22:44 GMT+08:00 Ted Yu :

> +1 on letting HBase-TRUNK jenkins show coverage report.
>
> Cheers
>
> On Sun, Mar 22, 2015 at 5:51 AM, 张铎  wrote:
>
> > Going to change the config of HBase-TRUNK jenkins to show findbugs,
> > checkstyle and jacoco coverage report.
> > The config has been tested on
> > https://builds.apache.org/job/HBase-TRUNK-jacoco for nearly 30 times.
> Not
> > much different from HBase-TRUNK unless it runs ~30% slower(the overhead
> of
> > collecting information for code coverage).
> > Thanks.
> >
> > 2015-03-12 5:08 GMT+08:00 Andrew Purtell :
> >
> > > +1, thanks a lot for improving our build hygiene.
> > >
> > > On Wed, Mar 11, 2015 at 2:01 PM, Enis Söztutar 
> > wrote:
> > >
> > > > Thanks Sean. This is great.
> > > >
> > > > Enis
> > > >
> > > > On Wed, Mar 11, 2015 at 1:54 PM, Sean Busbey 
> > > wrote:
> > > >
> > > > > FYI I just finished chasing down the breakage for mvn site on all
> > patch
> > > > > builds.
> > > > >
> > > > > HBASE-13191 consolidates the few places in the test-patch code
> where
> > we
> > > > > hard coded MAVEN_OPTS.
> > > > >
> > > > > If you look at the PreCommit job now, we use the "set environment
> > > > > variables" option to set MAVEN_OPTS and then everything else
> respects
> > > > that
> > > > > setting.
> > > > >
> > > > > I set the initial value to be a combination of the memory
> limitations
> > > > we've
> > > > > been actually running with (the ~6G was getting ignored) and the
> > > permgen
> > > > > needed for site.
> > > > >
> > > > > MAVEN_OPTS=-Xmx3100M -XX:-UsePerfData -XX:MaxPermSize=256m
> > > > >
> > > > > On Mon, Feb 23, 2015 at 11:46 PM, Stack  wrote:
> > > > >
> > > > > > I just made TRUNK and branch-1 builds use same jvm as patch-build
> > > > > > (hadoopqa) -- i.e. jdku51 -- and I set the MAVEN_OPTS to be the
> > same
> > > as
> > > > > > those of trunk build too, setting MAVEN_OPTS="-Xmx6100m"... it
> had
> > > been
> > > > > > 3000.
> > > > > >
> > > > > > Yours,
> > > > > > St.Ack
> > > > > >
> > > > > > On Wed, Dec 31, 2014 at 4:22 PM, Stack  wrote:
> > > > > >
> > > > > > > I upped hadoopqa retention to keep last 100 builds and or last
> 7
> > > > days,
> > > > > > > whichever comes first.
> > > > > > > St.Ack
> > > > > > >
> > > > > > > On Tue, Nov 4, 2014 at 9:38 AM, Stack 
> wrote:
> > > > > > >
> > > > > > >> Branch-1 and master have stabilized and now run mostly blue
> > (give
> > > or
> > > > > > take
> > > > > > >> the odd failure) [1][2]. Having a mostly blue branch-1 has
> > helped
> > > us
> > > > > > >> identify at least one destabilizing commit in the last few
> days,
> > > > maybe
> > > > > > two;
> > > > > > >> this is as it should be (smile).
> > > > > > >>
> > > > > > >> Lets keep our builds blue. If you commit a patch, make sure
> > > > subsequent
> > > > > > >> builds stay blue. You can subscribe to
> bui...@hbase.apache.org
> > to
> > > > get
> > > > > > >> notice of failures if not already subscribed.
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> St.Ack
> > > > > > >>
> > > > > > >> 1.
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.0/
> > > > > > >> 2.
> > https://builds.apache.org/view/H-L/view/HBase/job/HBase-TRUNK/
> > > > > > >

Re: [ANNOUNCE] Sean Busbey joins the Apache HBase PMC

[张铎]

Congratulations!

2015-03-27 13:12 GMT+08:00 Rajeshbabu Chintaguntla :

> Congratulations Sean!
>
> Thanks,
> Rajeshbabu.
>
> On Fri, Mar 27, 2015 at 10:28 AM, liushaohui 
> wrote:
>
> > Congratulations, Sean!
> >
> > -Shaohui Liu
> >
> >
> > On 03/27/2015 12:55 PM, Jerry He wrote:
> >
> >> Congratulations!!
> >> On Mar 26, 2015 8:41 PM, "Pankaj kr"  wrote:
> >>
> >>  Congrats Sean...!!
> >>>
> >>> Regards,
> >>> Pankaj
> >>> -Original Message-
> >>> From: Ted Yu [mailto:yuzhih...@gmail.com]
> >>> Sent: 27 March 2015 08:29
> >>> To: dev@hbase.apache.org; lars hofhansl
> >>> Cc: u...@hbase.apache.org
> >>> Subject: Re: [ANNOUNCE] Sean Busbey joins the Apache HBase PMC
> >>>
> >>> Congratulations, Sean.
> >>>
> >>> On Thu, Mar 26, 2015 at 2:49 PM, lars hofhansl
> >>>  >>> wrote:
> >>>
> >>>  Yeah! :)
> From: Andrew Purtell 
>    To: "u...@hbase.apache.org" ; "
>  dev@hbase.apache.org" 
>    Sent: Thursday, March 26, 2015 10:26 AM
>    Subject: [ANNOUNCE] Sean Busbey joins the Apache HBase PMC
> 
>  On behalf of the Apache HBase PMC I"m pleased to announce that Sean
>  Busbey has accepted our invitation to become a PMC member on the
>  Apache HBase project. Sean has been an active and positive contributor
>  in many areas, including on project meta-concerns such as versioning,
>  build infrastructure, code reviews, etc. He's a natural and we're
>  looking forward to many more future contributions.
> 
>  Welcome to the PMC, Sean!
> 
>  --
>  Best regards,
> 
> - Andy
> 
>  Problems worthy of attack prove their worth by hitting back. - Piet
>  Hein (via Tom White)
> 
> 
> 
> 
> 
> >
>

Re: Testing and CI -- Apache Jenkins Builds (WAS -> Re: Testing)

[张铎]

Going to change the config of HBase-TRUNK jenkins to show findbugs,
checkstyle and jacoco coverage report.
The config has been tested on
https://builds.apache.org/job/HBase-TRUNK-jacoco for nearly 30 times. Not
much different from HBase-TRUNK unless it runs ~30% slower(the overhead of
collecting information for code coverage).
Thanks.

2015-03-12 5:08 GMT+08:00 Andrew Purtell :

> +1, thanks a lot for improving our build hygiene.
>
> On Wed, Mar 11, 2015 at 2:01 PM, Enis Söztutar  wrote:
>
> > Thanks Sean. This is great.
> >
> > Enis
> >
> > On Wed, Mar 11, 2015 at 1:54 PM, Sean Busbey 
> wrote:
> >
> > > FYI I just finished chasing down the breakage for mvn site on all patch
> > > builds.
> > >
> > > HBASE-13191 consolidates the few places in the test-patch code where we
> > > hard coded MAVEN_OPTS.
> > >
> > > If you look at the PreCommit job now, we use the "set environment
> > > variables" option to set MAVEN_OPTS and then everything else respects
> > that
> > > setting.
> > >
> > > I set the initial value to be a combination of the memory limitations
> > we've
> > > been actually running with (the ~6G was getting ignored) and the
> permgen
> > > needed for site.
> > >
> > > MAVEN_OPTS=-Xmx3100M -XX:-UsePerfData -XX:MaxPermSize=256m
> > >
> > > On Mon, Feb 23, 2015 at 11:46 PM, Stack  wrote:
> > >
> > > > I just made TRUNK and branch-1 builds use same jvm as patch-build
> > > > (hadoopqa) -- i.e. jdku51 -- and I set the MAVEN_OPTS to be the same
> as
> > > > those of trunk build too, setting MAVEN_OPTS="-Xmx6100m"... it had
> been
> > > > 3000.
> > > >
> > > > Yours,
> > > > St.Ack
> > > >
> > > > On Wed, Dec 31, 2014 at 4:22 PM, Stack  wrote:
> > > >
> > > > > I upped hadoopqa retention to keep last 100 builds and or last 7
> > days,
> > > > > whichever comes first.
> > > > > St.Ack
> > > > >
> > > > > On Tue, Nov 4, 2014 at 9:38 AM, Stack  wrote:
> > > > >
> > > > >> Branch-1 and master have stabilized and now run mostly blue (give
> or
> > > > take
> > > > >> the odd failure) [1][2]. Having a mostly blue branch-1 has helped
> us
> > > > >> identify at least one destabilizing commit in the last few days,
> > maybe
> > > > two;
> > > > >> this is as it should be (smile).
> > > > >>
> > > > >> Lets keep our builds blue. If you commit a patch, make sure
> > subsequent
> > > > >> builds stay blue. You can subscribe to bui...@hbase.apache.org to
> > get
> > > > >> notice of failures if not already subscribed.
> > > > >>
> > > > >> Thanks,
> > > > >> St.Ack
> > > > >>
> > > > >> 1. https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.0/
> > > > >> 2. https://builds.apache.org/view/H-L/view/HBase/job/HBase-TRUNK/
> > > > >>
> > > > >>
> > > > >> On Mon, Oct 13, 2014 at 4:41 PM, Stack  wrote:
> > > > >>
> > > > >>> A few notes on testing.
> > > > >>>
> > > > >>> Too long to read, infra is more capable now and after some work,
> we
> > > are
> > > > >>> seeing branch-1 and trunk mostly running blue. Lets try and keep
> it
> > > > this
> > > > >>> way going forward.
> > > > >>>
> > > > >>> Apache Infra has new, more capable hardware.
> > > > >>>
> > > > >>> A recent spurt of test fixing combined with more capable hardware
> > > seems
> > > > >>> to have gotten us to a new place; tests are mostly passing now on
> > > > branch-1
> > > > >>> and master.  Lets try and keep it this way and start to trust our
> > > test
> > > > runs
> > > > >>> again.  Just a few flakies remain.  Lets try and nail them.
> > > > >>>
> > > > >>> Our tests now run in parallel with other test suites where
> previous
> > > we
> > > > >>> ran alone. You can see this sometimes when our zombie detector
> > > reports
> > > > >>> tests from another project altogether as lingerers (To be fixed).
> > > > Some of
> > > > >>> our tests are failing because a concurrent hbase run is undoing
> > > > classes and
> > > > >>> data from under it. Also, lets fix.
> > > > >>>
> > > > >>> Our tests are brittle. It takes 75minutes for them to complete.
> > Many
> > > > >>> are heavy-duty integration tests starting up multiple clusters
> and
> > > > >>> mapreduce all in the one JVM. It is a miracle they pass at all.
> > > > Usually
> > > > >>> integration tests have been cast as unit tests because there was
> no
> > > > where
> > > > >>> else for them to get an airing.  We have the hbase-it suite now
> > which
> > > > would
> > > > >>> be a more apt place but until these are run on a regular basis in
> > > > public
> > > > >>> for all to see, the fat integration tests disguised as unit tests
> > > will
> > > > >>> remain.  A review of our current unit tests weeding the old cruft
> > and
> > > > the
> > > > >>> no longer relevant or duplicates would be a nice undertaking if
> > > > someone is
> > > > >>> looking to contribute.
> > > > >>>
> > > > >>> Alex Newman has been working on making our tests work up on
> travis
> > > and
> > > > >>> circle-ci.  That'll be sweet when it goes end-to-end.  He also
> > added
> > > in
> > > > >>> some "type" categoriza

Re: I'm currently modifying the jenkins configuration

[张铎]

Thanks, let me copy a new job to debug and switch the config of TRUNK back.

2015-03-17 22:37 GMT+08:00 Sean Busbey :

> I'm available to help as well.
>
> In the future, copy the job and change the notification settings. That way
> you can get things working in parallel and then switch over when it does
> what you want.
>
> --
> Sean
> On Mar 17, 2015 8:01 AM, "Ted Yu"  wrote:
>
> > Please let me know in case you need any help.
> >
> > Thanks
> >
> >
> >
> > > On Mar 17, 2015, at 1:19 AM, 张铎  wrote:
> > >
> > > Try to show jacoco coverage report.
> > >
> > > So do not worry about the "Integrated in HBase-TRUNK" error message
> > posted
> > > on jira.
> > >
> > > I just abort a build. Or maybe more if it it does not work.
> > >
> > > I will finish this ASAP.
> > >
> > > Thanks.
> >
>

I'm currently modifying the jenkins configuration

[张铎]

Try to show jacoco coverage report.

So do not worry about the "Integrated in HBase-TRUNK" error message posted
on jira.

I just abort a build. Or maybe more if it it does not work.

I will finish this ASAP.

Thanks.

Re: feature request and question: "BigPut" and "BigGet"

[张铎]

If LOB means data larger than 10MB or even 100MB, why not just use an
FileSystem instead of HBase?
For a FileSystem it already has the stream interface...

2015-03-09 10:55 GMT+08:00 Wilm Schumacher :

> Hi,
>
> I have an idea for a feature in hbase which directly derives from the
> idea of the MOB feature. As Jonathan Hsieh pointed out, the only thing
> that limiting the feature to MOBs instead to LOBs is the memory
> allocation on client and server side. However, the "LOB feature" would
> be very handy for me and I think for some other users, too. Furthermore
> the fast fetching small files problem could be solved.
>
> The natural solution would be a "BigPut" and a "BigGet" class, which
> encounter that problem, which are capable of dealing with large amount
> of data without using too much memory. My plan by now is to creates
> classes that do e.g.
> BigPut BigPut.add( byte[] , byte[] , inputstream )
> and
> outputstream BigResult.value( byte[] , byte[] )
> (in addition to the normal byte[] to byte[] member functions)
>
> and pass the inputstreams through the AsyncProcess class to the RPC or
> in reverse the outputstream for the BigResult class. By this plan the
> client and server would have to throw out some threads to deal with
> multiple streams[1].
>
> By now I dig into the hbase-client (2.0.0) sources and I think that my
> plan would be quite invasive to the existing code ... but is doable.
> However, regarding the very open development model of hbase features I
> think it could be adressed.
>
> But I'm vry new to hbase development and just started to read the
> source. Before I dig to deep into the problem I wanted to ask here if
> there is any show stopper I'm missing by now?
> To make a list of questions for that feature:
> * As this plan probably won't break the thread model of the
> hbase-client, is there any problem on the (region) server side? Or is
> there any blocking/race condition problem elsewhere I miss by now?
> * Is it a bad plan to pump several 100s of MB through one RPC in a
> separate thread? If yes ... why?
> * Are there any other fundamental problems I miss by now which makes
> that a horrible plan?
> * Is there already some dev onging? I didn't found something on jira.
> But that doesn't mean anything :/
> * Does anyone have a better name than "BigPut" :D?
>
> And at last:
> * Is it a better plan to create a separate "MOB/LOB service"?[2]
>
> Best wishes
>
> Wilm
>
> [1] or one could limit the number of streams to one. By this the
> threading problem would be much more simple to encounter as only one
> "RPC" would be neccessary.
>
> [2] on one hand it is easier to bare LOBs in mind if you create a
> service e.g. with a rest interface (multipart data etc), on the other
> hand you have to reinvent the wheel (compaction etc.)
>

Re: [VOTE] Sixth release candidate for HBase 1.0.0 (RC5) is available. Please vote by Feb 19 2015

[张铎]

See HBASE-13070, I found the problem finally. It is a testcase issue so I
do not think it should block a release.

2015-02-20 10:12 GMT+08:00 Andrew Purtell :

> Maybe its just me
>
>
> On Thu, Feb 19, 2015 at 11:10 AM, Enis Söztutar 
> wrote:
>
> > I did not run the unit tests locally. I was relying on the following
> build
> > which is 100% pass:
> >
> > https://builds.apache.org/view/All/job/HBase-1.0/746/
> >
> > It has the exact RC5 bits (6c98bff7b719efdb16f71606f3b7d8229445eb81). Let
> > me kick a local UT run just in case.
> >
> > Enis
> >
> > On Thu, Feb 19, 2015 at 9:05 AM, Andrew Purtell 
> > wrote:
> >
> > > That list is missing unit testing. Did you happen to do that when
> > building
> > > but not mention it Enis? If so did they all pass for you?
> > >
> > > On Wed, Feb 18, 2015 at 7:48 PM, Enis Söztutar 
> > wrote:
> > >
> > > > Here is my RC5 testing so far:
> > > >
> > > >  - checked checksums, sigs
> > > >  - checked the bin and src artifacts
> > > >  - checked layouts
> > > >  - checked java files in src tarball, and jar files in bin tarball
> > > >  - checked the book and the site (they are new style)
> > > >  - checked javadocs for both devapi and userapi
> > > >  - checked reported version, build time, revision
> > > >  - run some smoke tests using shell
> > > >  - started local mode
> > > >  - run LTT local mode
> > > >  - checked the webUIs of master and region servers
> > > >  - checked JMX dump and debug dump
> > > >  - Build src with hadoop versions  2.2.0 2.3.0 2.4.0 2.4.1 2.5.0
> 2.5.1
> > > > 2.5.2 2.6.0
> > > > - Deployed at a 6 node cluster with Hadoop-2.6.0
> > > > - Run LTT over tables with NONE, DIFF, FAST_DIFF, and PREFIX encoding
> > > > - Run LTT over tables with NONE, GZ, LZO, LZ4 and SNAPPY compression
> > > > - Tested with Bucket cache with 500M off heap
> > > >
> > > > I am still running larger scale tests, and some CM tests on the
> > cluster.
> > > > Will report back tomorrow and cast my vote.
> > > >
> > > > Enis
> > > >
> > > >
> > > > On Wed, Feb 18, 2015 at 4:47 PM, 张铎  wrote:
> > > >
> > > > > TestCacheOnWrite itself has some problems. It
> > > > > uses TestHFileWriterV2.randomOrderedKey to generate a random byte
> > > array,
> > > > > then use first 32 bytes as row and other parts as family and
> > qualifier,
> > > > but
> > > > > TestHFileWriterV2.randomOrderedKey may return a byte array only
> > > contains
> > > > 32
> > > > > bytes, so there will be family and qualifier with zero length.
> > > > >
> > > > > I do not know if this is the reason why this test is flaky since it
> > > use a
> > > > > Random with pre-defined seed so the random sequence should be
> > stable. I
> > > > can
> > > > > modify the KeyValue generation part to see if it helps.
> > > > >
> > > > > BTW, the name 'randomOrderedKey' is ambiguous, may change to '
> > > > > randomOrderedRow'?
> > > > >
> > > > > 2015-02-19 7:11 GMT+08:00 Andrew Purtell :
> > > > >
> > > > > > I'm not able to get a clean unit test run when building from
> source
> > > > using
> > > > > > 7u67. TestCacheOnWrite and TestSplitLogManager fail for me, maybe
> > > more
> > > > > but
> > > > > > the build doesn't get past hbase-server. Maybe these are known
> > > issues?
> > > > If
> > > > > > not I'll dig in when I get some time.
> > > > > >
> > > > > >
> > > > > > On Sat, Feb 14, 2015 at 9:55 PM, Enis Söztutar 
> > > > wrote:
> > > > > >
> > > > > > > It gives me great pleasure to announce that the sixth release
> > > > candidate
> > > > > > for
> > > > > > > the release
> > > > > > > 1.0.0 (HBase-1.0.0RC5), is available for download at
> > > > > > > https://dist.apache.org/repos/dist/dev/hbase/hbase-1.0.0RC5/
> > > > > > >
> > > > > > > Maven artifacts are also available in the temporary repository
> >

Re: [VOTE] Sixth release candidate for HBase 1.0.0 (RC5) is available. Please vote by Feb 19 2015

[张铎]

TestCacheOnWrite itself has some problems. It
uses TestHFileWriterV2.randomOrderedKey to generate a random byte array,
then use first 32 bytes as row and other parts as family and qualifier, but
TestHFileWriterV2.randomOrderedKey may return a byte array only contains 32
bytes, so there will be family and qualifier with zero length.

I do not know if this is the reason why this test is flaky since it use a
Random with pre-defined seed so the random sequence should be stable. I can
modify the KeyValue generation part to see if it helps.

BTW, the name 'randomOrderedKey' is ambiguous, may change to '
randomOrderedRow'?

2015-02-19 7:11 GMT+08:00 Andrew Purtell :

> I'm not able to get a clean unit test run when building from source using
> 7u67. TestCacheOnWrite and TestSplitLogManager fail for me, maybe more but
> the build doesn't get past hbase-server. Maybe these are known issues? If
> not I'll dig in when I get some time.
>
>
> On Sat, Feb 14, 2015 at 9:55 PM, Enis Söztutar  wrote:
>
> > It gives me great pleasure to announce that the sixth release candidate
> for
> > the release
> > 1.0.0 (HBase-1.0.0RC5), is available for download at
> > https://dist.apache.org/repos/dist/dev/hbase/hbase-1.0.0RC5/
> >
> > Maven artifacts are also available in the temporary repository
> > https://repository.apache.org/content/repositories/orgapachehbase-1065
> >
> > Signed with my code signing key E964B5FF. Can be found here:
> > https://people.apache.org/keys/committer/enis.asc
> >
> >  Signed tag in the repository can be found here:
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=hbase.git;a=tag;h=c4660912e9b46c917a9aba2106be4bf74182a764
> >
> > HBase 1.0.0 is the next stable release, and the start of "semantic
> > versioned"
> > releases (See [1]).
> >
> > The theme of 1.0.0 release is to become a stable base for future 1.x
> series
> > of releases. We aim to achieve at least the same level of stability of
> 0.98
> > releases.
> >
> > 1.0.0 release contains 202 fixes on top of 0.99.2 release. Together with
> > the
> > previous 0.99.x releases, major changes in 1.0.0 are listed (but not
> > limited to)
> > below. Note that all previous 0.99.x releases are developer preview
> > releases, and will
> > NOT be supported in any form.
> >
> > API Cleanup and changes
> >   1.0.0 introduces new APIs, and deprecates some of commonly-used
> >   client side APIs (HTableInterface, HTable and HBaseAdmin).
> >   We advise to update your application to use the new style of APIs,
> since
> >   deprecated APIs might be removed in future releases (2.x). See [2] and
> > [3]
> >   for an overview of changes. All Client side API's are marked with
> >   InterfaceAudience.Public class, indicating that the class/method is an
> >   official "client API" for HBase. All 1.x releases are planned to be API
> >   compatible for these classes. See [1] for an overview.
> >
> > Master runs a Region Server as well
> >   Starting with 1.0.0, the HBase master server and backup master servers
> > will
> >   also act as a region server. RPC port and info port for web UI is
> shared
> > for
> >   the master and region server roles. Active master can host regions of
> >   defined tables if configured (disabled by default). Backup masters will
> > not
> >   host regions.
> >
> > Read availability using timeline consistent region replicas
> >   This release contains Phase 1 items for experimental "Read availability
> > using
> >   timeline consistent region replicas" feature. A region can be hosted in
> >   multiple region servers in read-only mode. One of the replicas for the
> > region
> >   will be primary, accepting writes, and other replicas will be sharing
> the
> > same
> >   data files. Read requests can be done against any replica for the
> region
> > with
> >   backup RPCs for high availability with timeline consistency guarantees.
> > More
> >   information can be found at HBASE-10070.
> >
> > Online config change and other forward ports from 0.89-fb branch
> >   HBASE-12147 forward ported online config change which enables some of
> the
> >   configuration from the server to be reloaded without restarting the
> > region
> >   servers.
> >
> > Other notable improvements in 1.0.0 (including previous 0.99.x) are
> >  - A new web skin in time for 1.0 (http://hbase.apache.org)
> >  - Automatic tuning of global memstore and block cache sizes
> >  - Various security, tags and visibility labels improvements
> >  - Bucket cache improvements (usability and compressed data blocks)
> >  - A new pluggable replication endpoint to plug in to HBase's
> inter-cluster
> >replication to replicate to a custom data store
> >  - A Dockerfile to easily build and run HBase from source
> >  - Truncate table command
> >  - Region assignment to use hbase:meta table instead of zookeeper for
> > faster
> >region assignment (disabled by default)
> >  - Extensive documentation improvements
> >  - [HBASE-12511] - namespace permissions - add support from table
> creat

Re: com/google/protobuf/Message not found Exception

[张铎]

I'm not familiar with Karaf so I do not know what is a "feature" in Karaf.

But hbase-client depends on many other jars, so usually you need to deploy
them all, not only the hbase-client.jar and hadoop.jar. And for java
programs, some people use maven-shade-plugin to pack a big jar with all
dependencies in it and deploy it.

Hope this could help.

2015-02-06 8:38 GMT+08:00 :

> Dell Customer Communication
>
> Hi All,
>
> I'm getting   NoClassFound exception for /com/google/protobuf/Message
> while getting connection to HTable in HBase. I am running HBase server
> single node on a VM with HBaseClient running on OSGI Karaf. I'm using the
> HBase 0.94.15 and Hadoop 1.0.4. I wrapped these two jars as feature and
> deployed on Karaf.
>
> Any idea why this exception is thrown on Karaf? Which jar HBase 0.94.15 is
> using for this class? Do I need to deploy anything extra on Karaf?
>
> Thanks,
>
> YuLing
>
> Caused by: java.lang.ClassNotFoundException: com.google.protobuf.Message
>
> at
> org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader
> .java:501)
>
> at
> org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:42
> 1)
>
> at
> org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:41
> 2)
>
> at
> org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultCl
> assLoader.java:107)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>
> ... 64 more
>
>
>

Re: RS disk capacity limits

[张铎]

I think 10 disks each 2TB is better.
A disk can only seek about 50 times per second no matter it is 10TB or 2TB,
so more disks means you can do more seeks and increase the random read
write performance.

2014-12-13 9:04 GMT+08:00 Krishna :
>
> Hi,
>
> Is there any logical/practical limit on HBase RS storage size?
> Which works better for HBase - a region server with 10 disks that are each
> 2 TB or 2 disks that are each 10TB?
> I remember, one of the recommendations is to keep each disk on RS to be
> less than 6 TB - is that correct?
>
> Thanks
>

Re: Not able to create a table in a single node set up with trunk

[张铎]

Oh I think the second line is confused...

I mean the value of hbase.regionserver.port in master's hbase-site.xml
should be different with the value in regionserver's hbase-site.xml

and the value of hbase.regionserver.info.port in master's hbase-site.xml
should also be different with the value in regionserver's hbase-site.xml

Sorry for my English...

2014-11-07 21:55 GMT+08:00 张铎 :

> You need to use different hbase-site.xml for master and regionserver,
> which means you should make two install directories, one for master and one
> for regionserver.
>
> The main reason is hbase.regionserver.port
> and hbase.regionserver.info.port must be different(otherwise you will get
> "java.net.BindException: Address already in use")...
>
> I tried just now and it works. 26020 is the system regionserver which is
> started in the same process of master, 36020 is a regionserver which is
> started on the same machine.
>
> ServerNameStart timeRequests Per SecondNum. Regions
> hdb177.hy01,26020,1415246110120 <http://hdb177.hy01:26030/rs-status>Thu
> Nov 06 11:55:10 CST 201402hdb177.hy01,36020,1415367654355
> <http://hdb177.hy01:36030/rs-status>Fri Nov 07 21:40:54 CST 201401Total:20
> 3
>
> 2014-11-07 14:09 GMT+08:00 ramkrishna vasudevan <
> ramkrishna.s.vasude...@gmail.com>:
>
>> Hi
>>
>> In the current trunk version, where we have the master acting itself as a
>> region server does not allow to assign any usertables in a single node
>> setup.
>>
>> Is it by design now so that master table only hosts SYSTEM tables?
>>
>> So a single RS node setup would mean that the master and RS should be on
>> different nodes only.
>>
>> Regards
>> Ram
>>
>
>

Re: Not able to create a table in a single node set up with trunk

[张铎]

You need to use different hbase-site.xml for master and regionserver, which
means you should make two install directories, one for master and one for
regionserver.

The main reason is hbase.regionserver.port and hbase.regionserver.info.port
must be different(otherwise you will get "java.net.BindException: Address
already in use")...

I tried just now and it works. 26020 is the system regionserver which is
started in the same process of master, 36020 is a regionserver which is
started on the same machine.

ServerNameStart timeRequests Per SecondNum. Regions
hdb177.hy01,26020,1415246110120 Thu Nov
06 11:55:10 CST 201402hdb177.hy01,36020,1415367654355
Fri Nov 07 21:40:54 CST 201401Total:203

2014-11-07 14:09 GMT+08:00 ramkrishna vasudevan <
ramkrishna.s.vasude...@gmail.com>:

> Hi
>
> In the current trunk version, where we have the master acting itself as a
> region server does not allow to assign any usertables in a single node
> setup.
>
> Is it by design now so that master table only hosts SYSTEM tables?
>
> So a single RS node setup would mean that the master and RS should be on
> different nodes only.
>
> Regards
> Ram
>