NOTICE: Nice testimony on benefits of the offheap read-path work now up on our blog

[Stack]

See writeup on how work done by your Yu Li, Sun Yu, Anoop Sam John, and
Ramkrishna S Vasudevan improved throughput at scale on Singles Day 2016 @
Alibaba.

Read all about it at https://blogs.apache.org/hbase/

Yours,
St.Ack

[jira] [Created] (HBASE-17762) Add logging to HBaseAdmin for user initiated tasks

[churro morales (JIRA)]

churro morales created HBASE-17762:
--

 Summary: Add logging to HBaseAdmin for user initiated tasks
 Key: HBASE-17762
 URL: https://issues.apache.org/jira/browse/HBASE-17762
 Project: HBase
  Issue Type: Task
Reporter: churro morales
Assignee: churro morales
 Fix For: 2.0.0, 1.4.0, 0.98.25


Things like auditing a forced major compaction are really useful and right now 
there is no logging when this is triggered.  Other actions may require logging 
as well. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Moving 2.0 forward

[Jerry He]

Thanks for the write-up, Stack.

I take the liberty to make the Spark item as:

4.3 hbase-spark


4.3.1 Status=

IN_PROGRESS

4.3.2 Owner=
Jerry
and team

We will see how much we can dot to fix-up and improve.

Thanks.

Jerry

Re: [VOTE] Backup/Restore feature for HBase 2.0, vote closing 3/11/2017

[Vladimir Rodionov]

No problem, we can extend deadline.

On Wed, Mar 8, 2017 at 3:31 PM, Ted Yu  wrote:

> March 11th is on weekend.
>
> Do you want to give people who haven't looked at the mega patch in depth
> some more time ?
>
> Cheers
>
> On Wed, Mar 8, 2017 at 1:32 PM, Vladimir Rodionov 
> wrote:
>
> > Hello, HBase folks
> >
> > For your consideration today is Backup/Restore feature for Apache HBAse
> > 2.0.
> > Backup code is available as a mega patch in HBASE-14123 (v61), applies
> > cleanly to the current master, all test PASS, patch has no other issues.
> >
> > The patch has gone through numerous rounds of code reviews and has
> probably
> > the most lengthy discussion thread on Apache JIRA (HBASE-14123) :)
> >
> > The work has been split into 3 phases (HBASE-14030, 14123, 14414) Two
> first
> > are complete, third one is still in progress.
> >
> >
> > *** Summary of work HBASE-14123
> >
> > The new feature introduces new command-line extensions to the hbase
> command
> > and, from the client side, is accessible through command-line only
> > Operations:
> > * Create full backup on a list of tables or backup set
> > * Create incremental backup image for table list or backup set
> > * Restore list of tables from a given backup image
> > * Show current backup progress
> > * Delete backup image and all related images
> > * Show history of backups
> > * Backup set operations: create backup set, add/remove table to/from
> backup
> > set, etc
> >
> > In the current implementation, the feature is already usable, meaning
> that
> > users can backup tables and restore them using provided command-line
> tools.
> > Both: full and incremental backups are supported.
> > This work is based on original work of IBM team (HBASE-7912). The full
> list
> > of JIRAs included in this mega patch can be found in three umbrella
> JIRAs:
> > HBASE-14030 (Phase 1), HBASE-14123 (Phase 2) and HBASE-14414 (Phase 3 -
> all
> > resolved ones made it into the patch)
> >
> > *** What are the remaining work items
> >
> > All remaining items can be found in Phase 3 umbrella JIRA: HBASE-14414.
> > They are split into 3 groups: BLOCKER, CRITICAL, MAJOR
> > Only BLOCKERs and CRITICALs are guaranteed for HBase 2.0 release.
> >
> > * BLOCKER
> >
> > * HBASE-14417 Incremental backup and bulk loading ( Patch available)
> > * HBASE-14135 HBase Backup/Restore Phase 3: Merge backup images
> > * HBASE-14141 HBase Backup/Restore Phase 3: Filter WALs on backup to
> > include only edits from backup tables (Patch available)
> > * HBASE-17133 Backup documentation
> > * HBASE-15227 Fault tolerance support
> >
> > * CRITICAL
> >
> > * HBASE-16465 Disable split/merges during backup
> >
> > We have umbrella JIRA (HBASE-14414) to track all the remaining work
> > All the BLOCKER and CRITICAL JIRAs currently in open state will be
> > implemented by 2.0 release time. Some MAJOR too, but it depends on
> resource
> > availability
> > The former development branch (HBASE-7912) is obsolete and will be
> > closed/deleted after the merge.
> > We want backup to be a GA feature in 2.0
> > We are going to support full backward compatibility for backup tool in
> 2.0
> > and onwards.
> >
> >  Configuration
> >
> > Backup is disabled, by default. To enable it, the following configuration
> > properties must be added to hbase-site.xml:
> >
> > hbase.backup.enable=true
> > hbase.master.logcleaner.plugins=YOUR_PLUGINS,org.
> > apache.hadoop.hbase.backup.master.BackupLogCleaner
> > hbase.procedure.master.classes=YOUR_CLASSES,org.
> > apache.hadoop.hbase.backup.master.LogRollMasterProcedureManager
> > hbase.procedure.regionserver.classes=YOUR_CLASSES,org.
> > apache.hadoop.hbase.backup.regionserver.LogRollRegionServerProcedureMa
> > nager
> >
> >
> > I would like to thank IBM team and Jerry He for original work,
> >
> > Enis, Ted, Stack, Matteo, Jerry for time spent on code reviews
> >
> > Special thanks to Ted Yu for his co-development work.
> >
> > References:
> >
> > https://issues.apache.org/jira/browse/HBASE-7912 (original IBM, contains
> > design doc)
> > https://issues.apache.org/jira/browse/HBASE-14030 (Phase 1)
> > https://issues.apache.org/jira/browse/HBASE-14123 (Phase 2)
> > https://issues.apache.org/jira/browse/HBASE-14414 (Phase 3)
> >
> > Please  vote +1/-1 by midnight Pacific Time (00:00
> > -0800 GMT) on March 11th  ​on whether or not we should merge this into
> the
> > current master.
> >
> > -Vladimir Rodionov
> >
>

Re: [VOTE] Backup/Restore feature for HBase 2.0, vote closing 3/11/2017

[Enis Söztutar]

Thanks Vladimir for the write up and the work. Glad to see progress.

Here is my +1. I'm pretty sure we can get the blockers in before the 2.0
timeframe with the momentum, so it is a good idea to merge now so that
development can continue in master, and there is more exposure for testing,
etc.

Enis

On Wed, Mar 8, 2017 at 1:32 PM, Vladimir Rodionov 
wrote:

> Hello, HBase folks
>
> For your consideration today is Backup/Restore feature for Apache HBAse
> 2.0.
> Backup code is available as a mega patch in HBASE-14123 (v61), applies
> cleanly to the current master, all test PASS, patch has no other issues.
>
> The patch has gone through numerous rounds of code reviews and has probably
> the most lengthy discussion thread on Apache JIRA (HBASE-14123) :)
>
> The work has been split into 3 phases (HBASE-14030, 14123, 14414) Two first
> are complete, third one is still in progress.
>
>
> *** Summary of work HBASE-14123
>
> The new feature introduces new command-line extensions to the hbase command
> and, from the client side, is accessible through command-line only
> Operations:
> * Create full backup on a list of tables or backup set
> * Create incremental backup image for table list or backup set
> * Restore list of tables from a given backup image
> * Show current backup progress
> * Delete backup image and all related images
> * Show history of backups
> * Backup set operations: create backup set, add/remove table to/from backup
> set, etc
>
> In the current implementation, the feature is already usable, meaning that
> users can backup tables and restore them using provided command-line tools.
> Both: full and incremental backups are supported.
> This work is based on original work of IBM team (HBASE-7912). The full list
> of JIRAs included in this mega patch can be found in three umbrella JIRAs:
> HBASE-14030 (Phase 1), HBASE-14123 (Phase 2) and HBASE-14414 (Phase 3 - all
> resolved ones made it into the patch)
>
> *** What are the remaining work items
>
> All remaining items can be found in Phase 3 umbrella JIRA: HBASE-14414.
> They are split into 3 groups: BLOCKER, CRITICAL, MAJOR
> Only BLOCKERs and CRITICALs are guaranteed for HBase 2.0 release.
>
> * BLOCKER
>
> * HBASE-14417 Incremental backup and bulk loading ( Patch available)
> * HBASE-14135 HBase Backup/Restore Phase 3: Merge backup images
> * HBASE-14141 HBase Backup/Restore Phase 3: Filter WALs on backup to
> include only edits from backup tables (Patch available)
> * HBASE-17133 Backup documentation
> * HBASE-15227 Fault tolerance support
>
> * CRITICAL
>
> * HBASE-16465 Disable split/merges during backup
>
> We have umbrella JIRA (HBASE-14414) to track all the remaining work
> All the BLOCKER and CRITICAL JIRAs currently in open state will be
> implemented by 2.0 release time. Some MAJOR too, but it depends on resource
> availability
> The former development branch (HBASE-7912) is obsolete and will be
> closed/deleted after the merge.
> We want backup to be a GA feature in 2.0
> We are going to support full backward compatibility for backup tool in 2.0
> and onwards.
>
>  Configuration
>
> Backup is disabled, by default. To enable it, the following configuration
> properties must be added to hbase-site.xml:
>
> hbase.backup.enable=true
> hbase.master.logcleaner.plugins=YOUR_PLUGINS,org.
> apache.hadoop.hbase.backup.master.BackupLogCleaner
> hbase.procedure.master.classes=YOUR_CLASSES,org.
> apache.hadoop.hbase.backup.master.LogRollMasterProcedureManager
> hbase.procedure.regionserver.classes=YOUR_CLASSES,org.
> apache.hadoop.hbase.backup.regionserver.LogRollRegionServerProcedureMa
> nager
>
>
> I would like to thank IBM team and Jerry He for original work,
>
> Enis, Ted, Stack, Matteo, Jerry for time spent on code reviews
>
> Special thanks to Ted Yu for his co-development work.
>
> References:
>
> https://issues.apache.org/jira/browse/HBASE-7912 (original IBM, contains
> design doc)
> https://issues.apache.org/jira/browse/HBASE-14030 (Phase 1)
> https://issues.apache.org/jira/browse/HBASE-14123 (Phase 2)
> https://issues.apache.org/jira/browse/HBASE-14414 (Phase 3)
>
> Please  vote +1/-1 by midnight Pacific Time (00:00
> -0800 GMT) on March 11th  ​on whether or not we should merge this into the
> current master.
>
> -Vladimir Rodionov
>

Re: [VOTE] Backup/Restore feature for HBase 2.0, vote closing 3/11/2017

[Ted Yu]

March 11th is on weekend.

Do you want to give people who haven't looked at the mega patch in depth
some more time ?

Cheers

On Wed, Mar 8, 2017 at 1:32 PM, Vladimir Rodionov 
wrote:

> Hello, HBase folks
>
> For your consideration today is Backup/Restore feature for Apache HBAse
> 2.0.
> Backup code is available as a mega patch in HBASE-14123 (v61), applies
> cleanly to the current master, all test PASS, patch has no other issues.
>
> The patch has gone through numerous rounds of code reviews and has probably
> the most lengthy discussion thread on Apache JIRA (HBASE-14123) :)
>
> The work has been split into 3 phases (HBASE-14030, 14123, 14414) Two first
> are complete, third one is still in progress.
>
>
> *** Summary of work HBASE-14123
>
> The new feature introduces new command-line extensions to the hbase command
> and, from the client side, is accessible through command-line only
> Operations:
> * Create full backup on a list of tables or backup set
> * Create incremental backup image for table list or backup set
> * Restore list of tables from a given backup image
> * Show current backup progress
> * Delete backup image and all related images
> * Show history of backups
> * Backup set operations: create backup set, add/remove table to/from backup
> set, etc
>
> In the current implementation, the feature is already usable, meaning that
> users can backup tables and restore them using provided command-line tools.
> Both: full and incremental backups are supported.
> This work is based on original work of IBM team (HBASE-7912). The full list
> of JIRAs included in this mega patch can be found in three umbrella JIRAs:
> HBASE-14030 (Phase 1), HBASE-14123 (Phase 2) and HBASE-14414 (Phase 3 - all
> resolved ones made it into the patch)
>
> *** What are the remaining work items
>
> All remaining items can be found in Phase 3 umbrella JIRA: HBASE-14414.
> They are split into 3 groups: BLOCKER, CRITICAL, MAJOR
> Only BLOCKERs and CRITICALs are guaranteed for HBase 2.0 release.
>
> * BLOCKER
>
> * HBASE-14417 Incremental backup and bulk loading ( Patch available)
> * HBASE-14135 HBase Backup/Restore Phase 3: Merge backup images
> * HBASE-14141 HBase Backup/Restore Phase 3: Filter WALs on backup to
> include only edits from backup tables (Patch available)
> * HBASE-17133 Backup documentation
> * HBASE-15227 Fault tolerance support
>
> * CRITICAL
>
> * HBASE-16465 Disable split/merges during backup
>
> We have umbrella JIRA (HBASE-14414) to track all the remaining work
> All the BLOCKER and CRITICAL JIRAs currently in open state will be
> implemented by 2.0 release time. Some MAJOR too, but it depends on resource
> availability
> The former development branch (HBASE-7912) is obsolete and will be
> closed/deleted after the merge.
> We want backup to be a GA feature in 2.0
> We are going to support full backward compatibility for backup tool in 2.0
> and onwards.
>
>  Configuration
>
> Backup is disabled, by default. To enable it, the following configuration
> properties must be added to hbase-site.xml:
>
> hbase.backup.enable=true
> hbase.master.logcleaner.plugins=YOUR_PLUGINS,org.
> apache.hadoop.hbase.backup.master.BackupLogCleaner
> hbase.procedure.master.classes=YOUR_CLASSES,org.
> apache.hadoop.hbase.backup.master.LogRollMasterProcedureManager
> hbase.procedure.regionserver.classes=YOUR_CLASSES,org.
> apache.hadoop.hbase.backup.regionserver.LogRollRegionServerProcedureMa
> nager
>
>
> I would like to thank IBM team and Jerry He for original work,
>
> Enis, Ted, Stack, Matteo, Jerry for time spent on code reviews
>
> Special thanks to Ted Yu for his co-development work.
>
> References:
>
> https://issues.apache.org/jira/browse/HBASE-7912 (original IBM, contains
> design doc)
> https://issues.apache.org/jira/browse/HBASE-14030 (Phase 1)
> https://issues.apache.org/jira/browse/HBASE-14123 (Phase 2)
> https://issues.apache.org/jira/browse/HBASE-14414 (Phase 3)
>
> Please  vote +1/-1 by midnight Pacific Time (00:00
> -0800 GMT) on March 11th  ​on whether or not we should merge this into the
> current master.
>
> -Vladimir Rodionov
>

Re: hbase build sizes

[Ted Yu]

As I commented in my previous email, there was no other change made.

On Wed, Mar 8, 2017 at 1:37 PM, Sean Busbey  wrote:

> Change is in place, I'll take a look at some build results tomorrow to
> see if it's going well.
>
> In the mean time, just a heads up to the committers that contributors
> will have questions if things fail and the logs didn't get properly
> zipped up.
>
> @Ted Yu I also undid your change of keeping builds for only 3 days.
> Please don't change the retention period; 3 days is not enough time
> for us reasonably expect volunteer contributors to circle back on
> getting their build results. If you updated the retention on any of
> the other jobs please restore them.
>
> On Wed, Mar 8, 2017 at 3:26 PM, Sean Busbey  wrote:
> > Okay, I'm reasonably sure this is just test logs. I'm updating our
> > builds to zip up the test logs as a post-build action, which should
> > bring down our space consumption by a couple of orders of magnitude.
> >
> >
> > On Wed, Mar 8, 2017 at 3:03 PM, Sean Busbey 
> wrote:
> >> I grabbed one of the items Chris pointed me towards:
> >>
> >> https://builds.apache.org/job/PreCommit-HBASE-Build/5991/
> >>
> >> If you download the archived artifacts as a zip, it's ~61MiB. If you
> >> uncompressed that zip file, you get ~1.3GiB of logs.
> >>
> >> I'm going to try to grab a couple more to make sure it's consistently
> test logs.
> >>
> >> On Wed, Mar 8, 2017 at 2:31 PM, Chris Lambertus  wrote:
> >>> Hi HBase folks,
> >>>
> >>> Just wanted to advise you of some build size issues we noticed on the
> jenkins master. It looks like over a span of about 2 weeks, the hbase
> builds are using a very large amount of storage in the retained builds
> directory:
> >>>
> >>> 79G PreCommit-HBASE-Build
> >>> 72G HBase-1.3-JDK8
> >>> 68G HBase-1.3-JDK7
> >>> 60G HBase-1.2-JDK7t
> >>> 59G HBase-1.2-JDK8
> >>> 51G HBase-1.2-IT
> >>> 36G HBase Website Link Ckecker
> >>>
> >>> Each stored build appears to use on the average of 600-1.1GB of disk
> space, and there are over 200 builds in two weeks for PreCommit-HBASE-Build
> alone. Some are much smaller (1-8MB.)
> >>>
> >>> We are running a bit low on storage on the jenkins master, so anything
> you could help us do to reduce the footprint here would be appreciated.
> >>>
> >>> I’ve already discussed this a bit with Sean Busbey on Hipchat, he
> asked me to follow up here.
> >>>
> >>> Thanks,
> >>>
> >>> -Chris
> >>> ASF Infra
> >>>
> >>
> >>
> >>
> >> --
> >> Sean
> >
> >
> >
> > --
> > Sean
>
>
>
> --
> Sean
>

[jira] [Created] (HBASE-17761) Test TestRemoveRegionMetrics.testMoveRegion fails intermittently because of race condition

[Umesh Agashe (JIRA)]

Umesh Agashe created HBASE-17761:


 Summary: Test TestRemoveRegionMetrics.testMoveRegion fails 
intermittently because of race condition
 Key: HBASE-17761
 URL: https://issues.apache.org/jira/browse/HBASE-17761
 Project: HBase
  Issue Type: Bug
  Components: Region Assignment
Reporter: Umesh Agashe
Assignee: Umesh Agashe


After moving the region code waits till all regions are assigned but not for 
region to go online on a destination server. On branch-1 TestAdmin1.java has a 
function moveRegionAndWait() that moves the region and waits for the region to 
become online on destination server. We need to use that function in this test.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: hbase build sizes

[Sean Busbey]

Change is in place, I'll take a look at some build results tomorrow to
see if it's going well.

In the mean time, just a heads up to the committers that contributors
will have questions if things fail and the logs didn't get properly
zipped up.

@Ted Yu I also undid your change of keeping builds for only 3 days.
Please don't change the retention period; 3 days is not enough time
for us reasonably expect volunteer contributors to circle back on
getting their build results. If you updated the retention on any of
the other jobs please restore them.

On Wed, Mar 8, 2017 at 3:26 PM, Sean Busbey  wrote:
> Okay, I'm reasonably sure this is just test logs. I'm updating our
> builds to zip up the test logs as a post-build action, which should
> bring down our space consumption by a couple of orders of magnitude.
>
>
> On Wed, Mar 8, 2017 at 3:03 PM, Sean Busbey  wrote:
>> I grabbed one of the items Chris pointed me towards:
>>
>> https://builds.apache.org/job/PreCommit-HBASE-Build/5991/
>>
>> If you download the archived artifacts as a zip, it's ~61MiB. If you
>> uncompressed that zip file, you get ~1.3GiB of logs.
>>
>> I'm going to try to grab a couple more to make sure it's consistently test 
>> logs.
>>
>> On Wed, Mar 8, 2017 at 2:31 PM, Chris Lambertus  wrote:
>>> Hi HBase folks,
>>>
>>> Just wanted to advise you of some build size issues we noticed on the 
>>> jenkins master. It looks like over a span of about 2 weeks, the hbase 
>>> builds are using a very large amount of storage in the retained builds 
>>> directory:
>>>
>>> 79GPreCommit-HBASE-Build
>>> 72GHBase-1.3-JDK8
>>> 68GHBase-1.3-JDK7
>>> 60GHBase-1.2-JDK7t
>>> 59GHBase-1.2-JDK8
>>> 51GHBase-1.2-IT
>>> 36GHBase Website Link Ckecker
>>>
>>> Each stored build appears to use on the average of 600-1.1GB of disk space, 
>>> and there are over 200 builds in two weeks for PreCommit-HBASE-Build alone. 
>>> Some are much smaller (1-8MB.)
>>>
>>> We are running a bit low on storage on the jenkins master, so anything you 
>>> could help us do to reduce the footprint here would be appreciated.
>>>
>>> I’ve already discussed this a bit with Sean Busbey on Hipchat, he asked me 
>>> to follow up here.
>>>
>>> Thanks,
>>>
>>> -Chris
>>> ASF Infra
>>>
>>
>>
>>
>> --
>> Sean
>
>
>
> --
> Sean



-- 
Sean

[jira] [Resolved] (HBASE-17751) Update snapshot manifest to include table it was created from

[Josh Elser (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser resolved HBASE-17751.

Resolution: Invalid

The table name is already included in the {{SnapshotDefinition}}. It looks like 
I just missed this the first time around...

> Update snapshot manifest to include table it was created from
> -
>
> Key: HBASE-17751
> URL: https://issues.apache.org/jira/browse/HBASE-17751
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Josh Elser
>
> In order to tie a snapshot back to a table (and add its utilization to that 
> table's quota), we need to store that lineage of source table to snapshot.
> Add that information to the snapshot manifest file in a manner which is 
> backwards compatible (e.g. upon upgrade, some snapshots may not have this 
> field -- need to make sure we can handle both cases safely).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: hbase build sizes

[Ted Yu]

For PreCommit-HBASE-Build, I have cut down days to keep builds from 14 to 3.

For HBase-1.3-JDK8, the fields is empty. Not sure what the default is.

"/configure" loaded vey slowly. I can try other builds later.

Cheers

On Wed, Mar 8, 2017 at 12:31 PM, Chris Lambertus  wrote:

> Hi HBase folks,
>
> Just wanted to advise you of some build size issues we noticed on the
> jenkins master. It looks like over a span of about 2 weeks, the hbase
> builds are using a very large amount of storage in the retained builds
> directory:
>
> 79G PreCommit-HBASE-Build
> 72G HBase-1.3-JDK8
> 68G HBase-1.3-JDK7
> 60G HBase-1.2-JDK7t
> 59G HBase-1.2-JDK8
> 51G HBase-1.2-IT
> 36G HBase Website Link Ckecker
>
> Each stored build appears to use on the average of 600-1.1GB of disk
> space, and there are over 200 builds in two weeks for PreCommit-HBASE-Build
> alone. Some are much smaller (1-8MB.)
>
> We are running a bit low on storage on the jenkins master, so anything you
> could help us do to reduce the footprint here would be appreciated.
>
> I’ve already discussed this a bit with Sean Busbey on Hipchat, he asked me
> to follow up here.
>
> Thanks,
>
> -Chris
> ASF Infra
>
>

[VOTE] Backup/Restore feature for HBase 2.0, vote closing 3/11/2017

[Vladimir Rodionov]

Hello, HBase folks

For your consideration today is Backup/Restore feature for Apache HBAse 2.0.
Backup code is available as a mega patch in HBASE-14123 (v61), applies
cleanly to the current master, all test PASS, patch has no other issues.

The patch has gone through numerous rounds of code reviews and has probably
the most lengthy discussion thread on Apache JIRA (HBASE-14123) :)

The work has been split into 3 phases (HBASE-14030, 14123, 14414) Two first
are complete, third one is still in progress.


*** Summary of work HBASE-14123

The new feature introduces new command-line extensions to the hbase command
and, from the client side, is accessible through command-line only
Operations:
* Create full backup on a list of tables or backup set
* Create incremental backup image for table list or backup set
* Restore list of tables from a given backup image
* Show current backup progress
* Delete backup image and all related images
* Show history of backups
* Backup set operations: create backup set, add/remove table to/from backup
set, etc

In the current implementation, the feature is already usable, meaning that
users can backup tables and restore them using provided command-line tools.
Both: full and incremental backups are supported.
This work is based on original work of IBM team (HBASE-7912). The full list
of JIRAs included in this mega patch can be found in three umbrella JIRAs:
HBASE-14030 (Phase 1), HBASE-14123 (Phase 2) and HBASE-14414 (Phase 3 - all
resolved ones made it into the patch)

*** What are the remaining work items

All remaining items can be found in Phase 3 umbrella JIRA: HBASE-14414.
They are split into 3 groups: BLOCKER, CRITICAL, MAJOR
Only BLOCKERs and CRITICALs are guaranteed for HBase 2.0 release.

* BLOCKER

* HBASE-14417 Incremental backup and bulk loading ( Patch available)
* HBASE-14135 HBase Backup/Restore Phase 3: Merge backup images
* HBASE-14141 HBase Backup/Restore Phase 3: Filter WALs on backup to
include only edits from backup tables (Patch available)
* HBASE-17133 Backup documentation
* HBASE-15227 Fault tolerance support

* CRITICAL

* HBASE-16465 Disable split/merges during backup

We have umbrella JIRA (HBASE-14414) to track all the remaining work
All the BLOCKER and CRITICAL JIRAs currently in open state will be
implemented by 2.0 release time. Some MAJOR too, but it depends on resource
availability
The former development branch (HBASE-7912) is obsolete and will be
closed/deleted after the merge.
We want backup to be a GA feature in 2.0
We are going to support full backward compatibility for backup tool in 2.0
and onwards.

 Configuration

Backup is disabled, by default. To enable it, the following configuration
properties must be added to hbase-site.xml:

hbase.backup.enable=true
hbase.master.logcleaner.plugins=YOUR_PLUGINS,org.apache.hadoop.hbase.backup.master.BackupLogCleaner
hbase.procedure.master.classes=YOUR_CLASSES,org.apache.hadoop.hbase.backup.master.LogRollMasterProcedureManager
hbase.procedure.regionserver.classes=YOUR_CLASSES,org.apache.hadoop.hbase.backup.regionserver.LogRollRegionServerProcedureManager


I would like to thank IBM team and Jerry He for original work,

Enis, Ted, Stack, Matteo, Jerry for time spent on code reviews

Special thanks to Ted Yu for his co-development work.

References:

https://issues.apache.org/jira/browse/HBASE-7912 (original IBM, contains
design doc)
https://issues.apache.org/jira/browse/HBASE-14030 (Phase 1)
https://issues.apache.org/jira/browse/HBASE-14123 (Phase 2)
https://issues.apache.org/jira/browse/HBASE-14414 (Phase 3)

Please  vote +1/-1 by midnight Pacific Time (00:00
-0800 GMT) on March 11th  ​on whether or not we should merge this into the
current master.

-Vladimir Rodionov

Re: hbase build sizes

[Sean Busbey]

Okay, I'm reasonably sure this is just test logs. I'm updating our
builds to zip up the test logs as a post-build action, which should
bring down our space consumption by a couple of orders of magnitude.


On Wed, Mar 8, 2017 at 3:03 PM, Sean Busbey  wrote:
> I grabbed one of the items Chris pointed me towards:
>
> https://builds.apache.org/job/PreCommit-HBASE-Build/5991/
>
> If you download the archived artifacts as a zip, it's ~61MiB. If you
> uncompressed that zip file, you get ~1.3GiB of logs.
>
> I'm going to try to grab a couple more to make sure it's consistently test 
> logs.
>
> On Wed, Mar 8, 2017 at 2:31 PM, Chris Lambertus  wrote:
>> Hi HBase folks,
>>
>> Just wanted to advise you of some build size issues we noticed on the 
>> jenkins master. It looks like over a span of about 2 weeks, the hbase builds 
>> are using a very large amount of storage in the retained builds directory:
>>
>> 79GPreCommit-HBASE-Build
>> 72GHBase-1.3-JDK8
>> 68GHBase-1.3-JDK7
>> 60GHBase-1.2-JDK7t
>> 59GHBase-1.2-JDK8
>> 51GHBase-1.2-IT
>> 36GHBase Website Link Ckecker
>>
>> Each stored build appears to use on the average of 600-1.1GB of disk space, 
>> and there are over 200 builds in two weeks for PreCommit-HBASE-Build alone. 
>> Some are much smaller (1-8MB.)
>>
>> We are running a bit low on storage on the jenkins master, so anything you 
>> could help us do to reduce the footprint here would be appreciated.
>>
>> I’ve already discussed this a bit with Sean Busbey on Hipchat, he asked me 
>> to follow up here.
>>
>> Thanks,
>>
>> -Chris
>> ASF Infra
>>
>
>
>
> --
> Sean



-- 
Sean

Re: hbase build sizes

[Sean Busbey]

I grabbed one of the items Chris pointed me towards:

https://builds.apache.org/job/PreCommit-HBASE-Build/5991/

If you download the archived artifacts as a zip, it's ~61MiB. If you
uncompressed that zip file, you get ~1.3GiB of logs.

I'm going to try to grab a couple more to make sure it's consistently test logs.

On Wed, Mar 8, 2017 at 2:31 PM, Chris Lambertus  wrote:
> Hi HBase folks,
>
> Just wanted to advise you of some build size issues we noticed on the jenkins 
> master. It looks like over a span of about 2 weeks, the hbase builds are 
> using a very large amount of storage in the retained builds directory:
>
> 79GPreCommit-HBASE-Build
> 72GHBase-1.3-JDK8
> 68GHBase-1.3-JDK7
> 60GHBase-1.2-JDK7t
> 59GHBase-1.2-JDK8
> 51GHBase-1.2-IT
> 36GHBase Website Link Ckecker
>
> Each stored build appears to use on the average of 600-1.1GB of disk space, 
> and there are over 200 builds in two weeks for PreCommit-HBASE-Build alone. 
> Some are much smaller (1-8MB.)
>
> We are running a bit low on storage on the jenkins master, so anything you 
> could help us do to reduce the footprint here would be appreciated.
>
> I’ve already discussed this a bit with Sean Busbey on Hipchat, he asked me to 
> follow up here.
>
> Thanks,
>
> -Chris
> ASF Infra
>



-- 
Sean

hbase build sizes

[Chris Lambertus]

Hi HBase folks,

Just wanted to advise you of some build size issues we noticed on the jenkins 
master. It looks like over a span of about 2 weeks, the hbase builds are using 
a very large amount of storage in the retained builds directory:

79GPreCommit-HBASE-Build
72GHBase-1.3-JDK8
68GHBase-1.3-JDK7
60GHBase-1.2-JDK7t
59GHBase-1.2-JDK8
51GHBase-1.2-IT
36GHBase Website Link Ckecker

Each stored build appears to use on the average of 600-1.1GB of disk space, and 
there are over 200 builds in two weeks for PreCommit-HBASE-Build alone. Some 
are much smaller (1-8MB.)

We are running a bit low on storage on the jenkins master, so anything you 
could help us do to reduce the footprint here would be appreciated.

I’ve already discussed this a bit with Sean Busbey on Hipchat, he asked me to 
follow up here.

Thanks,

-Chris
ASF Infra



signature.asc
Description: Message signed with OpenPGP

[jira] [Resolved] (HBASE-17728) Create separate build target for Configuration classes

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HBASE-17728.

  Resolution: Fixed
Hadoop Flags: Reviewed

> Create separate build target for Configuration classes
> --
>
> Key: HBASE-17728
> URL: https://issues.apache.org/jira/browse/HBASE-17728
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 17728.v1.txt, 17728.v2.txt, 17728.v3.txt, 17728.v4.txt, 
> 17728.v5.txt, 17728.v6.txt, 17728.v7.txt, 17728.v8.txt, 17728.v9.txt
>
>
> User is in security module.
> When User::isSecurityEnabled() is added, we need to query Configuration for 
> security setting.
> However, this introduces a circular build dependency:
> BUILD FAILED: Cycle found: //connection:connection -> //security:security -> 
> //core:core -> //connection:connection
> This issue is to create separate build target for Configuration which is 
> depended upon by both core and security modules.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Successful: HBase Generate Website

[Apache Jenkins Server]

Build status: Successful

If successful, the website and docs have been generated. To update the live 
site, follow the instructions below. If failed, skip to the bottom of this 
email.

Use the following commands to download the patch and apply it to a clean branch 
based on origin/asf-site. If you prefer to keep the hbase-site repo around 
permanently, you can skip the clone step.

  git clone https://git-wip-us.apache.org/repos/asf/hbase-site.git

  cd hbase-site
  wget -O- 
https://builds.apache.org/job/hbase_generate_website/510/artifact/website.patch.zip
 | funzip > 58c76192bdbf1f4863c1c87d165c2e3b9674d4ad.patch
  git fetch
  git checkout -b asf-site-58c76192bdbf1f4863c1c87d165c2e3b9674d4ad 
origin/asf-site
  git am --whitespace=fix 58c76192bdbf1f4863c1c87d165c2e3b9674d4ad.patch

At this point, you can preview the changes by opening index.html or any of the 
other HTML pages in your local 
asf-site-58c76192bdbf1f4863c1c87d165c2e3b9674d4ad branch.

There are lots of spurious changes, such as timestamps and CSS styles in 
tables, so a generic git diff is not very useful. To see a list of files that 
have been added, deleted, renamed, changed type, or are otherwise interesting, 
use the following command:

  git diff --name-status --diff-filter=ADCRTXUB origin/asf-site

To see only files that had 100 or more lines changed:

  git diff --stat origin/asf-site | grep -E '[1-9][0-9]{2,}'

When you are satisfied, publish your changes to origin/asf-site using these 
commands:

  git commit --allow-empty -m "Empty commit" # to work around a current ASF 
INFRA bug
  git push origin asf-site-58c76192bdbf1f4863c1c87d165c2e3b9674d4ad:asf-site
  git checkout asf-site
  git branch -D asf-site-58c76192bdbf1f4863c1c87d165c2e3b9674d4ad

Changes take a couple of minutes to be propagated. You can verify whether they 
have been propagated by looking at the Last Published date at the bottom of 
http://hbase.apache.org/. It should match the date in the index.html on the 
asf-site branch in Git.

As a courtesy- reply-all to this email to let other committers know you pushed 
the site.



If failed, see https://builds.apache.org/job/hbase_generate_website/510/console

[jira] [Resolved] (HBASE-6941) LoadIncrementalHFiles uses the Tool interface incorrectly for loading configs

[Harsh J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J resolved HBASE-6941.

Resolution: Duplicate
  Assignee: (was: Harsh J)

> LoadIncrementalHFiles uses the Tool interface incorrectly for loading configs
> -
>
> Key: HBASE-6941
> URL: https://issues.apache.org/jira/browse/HBASE-6941
> Project: HBase
>  Issue Type: Bug
>  Components: Client
>Affects Versions: 0.90.6
>Reporter: Harsh J
> Attachments: HBASE-6941.patch
>
>
> The LoadIncrementalHFiles tool has pretty complex config loading structured 
> in it, which seems unnecessary and also causes problem since it is ignoring 
> any settings passed to it via Tool's -Dprop=value parameters.
> This makes integration with tools such as Oozie harder, as it doesn't accept 
> different addresses of ZK, etc. unless there's a hbase-site.xml on the 
> classpath to load from (which is painful to achieve on Oozie).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: HDFS Balancer

[Harsh J]

Hey Lars!,

I was on a similar line of investigation today, and I've filed
https://issues.apache.org/jira/browse/HBASE-17760 to change the text. The
pinning part of the text is relevant, but the command part isn't. In
addition, you'd need to manually use the FavoredNodeLoadBalancer work to
actually get HBase to apply pinning to its writes by passing around proper
favored-node hint hostnames. I've also linked past and future relevant work
JIRAs to that one.

Stumbled on this email when searching some follow-throughs, thought I'd
drop a note.

On Tue, 7 Mar 2017 at 20:18 Ted Yu  wrote:

> bq. how that - apparently wrong - information came about
>
> Maybe Sean / Misty can give some context.
>
> Cheers
>
> On Tue, Mar 7, 2017 at 6:37 AM, Lars George  wrote:
>
> > Hey Ted,
> >
> > Thanks Cpt. Obvious :)
> >
> > I know how to use "blame" or git log how to find the JIRA, but what I was
> > after is how that - apparently wrong - information came about. And if it
> is
> > wrong, what _is_ the current status of this feature.
> >
> > I do believe this is an important operational piece as it helps with
> > rearranging clusters. Since it seems to still be missing, I am wondering
> > what needs to be done here.
> >
> > Makes sense?
> >
> > Lars
> >
> > Sent from my iPhone
> >
> > > On 6 Mar 2017, at 19:50, Ted Yu  wrote:
> > >
> > > w.r.t. the first question, the quoted paragraph came from:
> > >
> > > HBASE-15332 Document how to take advantage of HDFS-6133 in HBase
> > >
> > >> On Mon, Mar 6, 2017 at 6:38 PM, Lars George 
> > wrote:
> > >>
> > >> Hi,
> > >>
> > >> I am trying to grok what came out of all these issues about the HDFS
> > >> balancer and being able to avoid it destroying HBase locality. There
> > >> is this https://issues.apache.org/jira/browse/HBASE-13021 from JM,
> and
> > >> the book http://hbase.apache.org/book.html#_hbase_and_hdfs refers to
> > >> https://issues.apache.org/jira/browse/HDFS-6133, stating:
> > >>
> > >> "HDFS-6133 provides the ability to exclude a given directory from the
> > >> HDFS load balancer, by setting the dfs.datanode.block-pinning.enabled
> > >> property to true in your HDFS configuration and running the following
> > >> hdfs command:
> > >>
> > >> $ sudo -u hdfs hdfs balancer -exclude /hbase"
> > >>
> > >> I checked the Balancer class in 2.7.2 and it does not have that
> > >> support, i.e. being able to exclude a path, it can only exclude hosts.
> > >> That is also clear from HDFS-6133, which adds favoured nodes, but not
> > >> being able to exclude paths (which would be nice).
> > >>
> > >> HBASE-13021 mentions that this works in tandem with the HBase favored
> > >> node feature, but that makes it much more complicated since you have
> > >> to pin individual regions to nodes, instead of doing that wholesale.
> > >>
> > >> Where does the above in the HBase book come from, and what is the
> > >> current state as far as you know?
> > >>
> > >> Cheers,
> > >> Lars
> > >>
> >
>

[jira] [Resolved] (HBASE-13021) Pin regions blocks to avoid HDFS balancing

[Harsh J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J resolved HBASE-13021.
-
   Resolution: Duplicate
Fix Version/s: (was: 2.0.0)

> Pin regions blocks to avoid HDFS balancing
> --
>
> Key: HBASE-13021
> URL: https://issues.apache.org/jira/browse/HBASE-13021
> Project: HBase
>  Issue Type: New Feature
>Reporter: Jean-Marc Spaggiari
>
> Now that HDFS-6133 is into HDFS, we might want to implement its features to 
> pin regions blocks to avoid them to be re-located.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HBASE-17760) HDFS Balancer doc is misleading

[Harsh J (JIRA)]

Harsh J created HBASE-17760:
---

 Summary: HDFS Balancer doc is misleading
 Key: HBASE-17760
 URL: https://issues.apache.org/jira/browse/HBASE-17760
 Project: HBase
  Issue Type: Bug
  Components: documentation
Reporter: Harsh J
Assignee: Harsh J
Priority: Minor


HBASE-15332 added a doc note about how to use HDFS-6133, but the steps it adds 
are incorrect. The specific balancer command provided in the doc note is 
incorrect and not required.

Since HBase uses favored nodes features internally (HBASE-7932), and HBASE-7942 
extended that information to cover HDFS hinting too, the only step required in 
the doc note is to enable the pinning feature DN-side.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HBASE-17759) TestComparators.testCellFieldsCompare fails on ppc64le

[Anup Halarnkar (JIRA)]

Anup Halarnkar created HBASE-17759:
--

 Summary:  TestComparators.testCellFieldsCompare fails on ppc64le
 Key: HBASE-17759
 URL: https://issues.apache.org/jira/browse/HBASE-17759
 Project: HBase
  Issue Type: Bug
  Components: integration tests
Affects Versions: 2.0.0
 Environment: OS: Ubuntu 14.04
Arch: Power (ppc64le)
Reporter: Anup Halarnkar
 Fix For: 2.0.0


Failed tests:
  TestComparators.testCellFieldsCompare:56 expected:<0> but was:<114>
Tests run: 133, Failures: 1, Errors: 0, Skipped: 1
--
Output of TestComparators-output.txt
2017-03-08 12:44:17,934 DEBUG [main] util.ClassSize(230): Using Unsafe to 
estimate memory layout
--
Output of TestComparators.txt
---
Test set: org.apache.hadoop.hbase.filter.TestComparators
---
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.153 sec <<< 
FAILURE! - in org.apache.hadoop.hbase.filter.TestComparators
testCellFieldsCompare(org.apache.hadoop.hbase.filter.TestComparators)  Time 
elapsed: 0.153 sec  <<< FAILURE!
java.lang.AssertionError: expected:<0> but was:<114>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:631)
at 
org.apache.hadoop.hbase.filter.TestComparators.testCellFieldsCompare(TestComparators.java:56)
---




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)