Re: [VOTE] Backup/Restore feature for HBase 2.0, vote closing 3/11/2017

[Vladimir Rodionov]

bump

On Thu, Mar 9, 2017 at 10:11 AM, Ted Yu  wrote:

> +1 from me as well.
>
> On Wed, Mar 8, 2017 at 3:48 PM, Enis Söztutar  wrote:
>
> > Thanks Vladimir for the write up and the work. Glad to see progress.
> >
> > Here is my +1. I'm pretty sure we can get the blockers in before the 2.0
> > timeframe with the momentum, so it is a good idea to merge now so that
> > development can continue in master, and there is more exposure for
> testing,
> > etc.
> >
> > Enis
> >
> > On Wed, Mar 8, 2017 at 1:32 PM, Vladimir Rodionov <
> vladrodio...@gmail.com>
> > wrote:
> >
> > > Hello, HBase folks
> > >
> > > For your consideration today is Backup/Restore feature for Apache HBAse
> > > 2.0.
> > > Backup code is available as a mega patch in HBASE-14123 (v61), applies
> > > cleanly to the current master, all test PASS, patch has no other
> issues.
> > >
> > > The patch has gone through numerous rounds of code reviews and has
> > probably
> > > the most lengthy discussion thread on Apache JIRA (HBASE-14123) :)
> > >
> > > The work has been split into 3 phases (HBASE-14030, 14123, 14414) Two
> > first
> > > are complete, third one is still in progress.
> > >
> > >
> > > *** Summary of work HBASE-14123
> > >
> > > The new feature introduces new command-line extensions to the hbase
> > command
> > > and, from the client side, is accessible through command-line only
> > > Operations:
> > > * Create full backup on a list of tables or backup set
> > > * Create incremental backup image for table list or backup set
> > > * Restore list of tables from a given backup image
> > > * Show current backup progress
> > > * Delete backup image and all related images
> > > * Show history of backups
> > > * Backup set operations: create backup set, add/remove table to/from
> > backup
> > > set, etc
> > >
> > > In the current implementation, the feature is already usable, meaning
> > that
> > > users can backup tables and restore them using provided command-line
> > tools.
> > > Both: full and incremental backups are supported.
> > > This work is based on original work of IBM team (HBASE-7912). The full
> > list
> > > of JIRAs included in this mega patch can be found in three umbrella
> > JIRAs:
> > > HBASE-14030 (Phase 1), HBASE-14123 (Phase 2) and HBASE-14414 (Phase 3 -
> > all
> > > resolved ones made it into the patch)
> > >
> > > *** What are the remaining work items
> > >
> > > All remaining items can be found in Phase 3 umbrella JIRA: HBASE-14414.
> > > They are split into 3 groups: BLOCKER, CRITICAL, MAJOR
> > > Only BLOCKERs and CRITICALs are guaranteed for HBase 2.0 release.
> > >
> > > * BLOCKER
> > >
> > > * HBASE-14417 Incremental backup and bulk loading ( Patch available)
> > > * HBASE-14135 HBase Backup/Restore Phase 3: Merge backup images
> > > * HBASE-14141 HBase Backup/Restore Phase 3: Filter WALs on backup to
> > > include only edits from backup tables (Patch available)
> > > * HBASE-17133 Backup documentation
> > > * HBASE-15227 Fault tolerance support
> > >
> > > * CRITICAL
> > >
> > > * HBASE-16465 Disable split/merges during backup
> > >
> > > We have umbrella JIRA (HBASE-14414) to track all the remaining work
> > > All the BLOCKER and CRITICAL JIRAs currently in open state will be
> > > implemented by 2.0 release time. Some MAJOR too, but it depends on
> > resource
> > > availability
> > > The former development branch (HBASE-7912) is obsolete and will be
> > > closed/deleted after the merge.
> > > We want backup to be a GA feature in 2.0
> > > We are going to support full backward compatibility for backup tool in
> > 2.0
> > > and onwards.
> > >
> > >  Configuration
> > >
> > > Backup is disabled, by default. To enable it, the following
> configuration
> > > properties must be added to hbase-site.xml:
> > >
> > > hbase.backup.enable=true
> > > hbase.master.logcleaner.plugins=YOUR_PLUGINS,org.
> > > apache.hadoop.hbase.backup.master.BackupLogCleaner
> > > hbase.procedure.master.classes=YOUR_CLASSES,org.
> > > apache.hadoop.hbase.backup.master.LogRollMasterProcedureManager
> > > hbase.procedure.regionserver.classes=YOUR_CLASSES,org.
> > > apache.hadoop.hbase.backup.regionserver.LogRollRegionServerProcedureMa
> > > nager
> > >
> > >
> > > I would like to thank IBM team and Jerry He for original work,
> > >
> > > Enis, Ted, Stack, Matteo, Jerry for time spent on code reviews
> > >
> > > Special thanks to Ted Yu for his co-development work.
> > >
> > > References:
> > >
> > > https://issues.apache.org/jira/browse/HBASE-7912 (original IBM,
> contains
> > > design doc)
> > > https://issues.apache.org/jira/browse/HBASE-14030 (Phase 1)
> > > https://issues.apache.org/jira/browse/HBASE-14123 (Phase 2)
> > > https://issues.apache.org/jira/browse/HBASE-14414 (Phase 3)
> > >
> > > Please  vote +1/-1 by midnight Pacific Time (00:00
> > > -0800 GMT) on March 11th  ​on whether or not we should merge this into
> > the
> > > current master.
> > >
> > > -Vladimir Rodionov
> > >
> >

[jira] [Resolved] (HBASE-17767) hbase CMS gc pause program

[Duo Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-17767.
---
Resolution: Invalid

> hbase CMS  gc  pause program
> 
>
> Key: HBASE-17767
> URL: https://issues.apache.org/jira/browse/HBASE-17767
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.1.2
>Reporter: gehaijiang
>
> cms gc pause serious program:
> 2017-03-10T10:15:29.524+0800: 4555920.160: [GC2017-03-10T10:15:29.524+0800: 
> 4555920.160: [ParNew: 3067133K->340736K(3067136K), 2.0586980 secs] 
> 80328431K->78058138K(100322560K), 2.0590280 secs] [Times: user=3.94 sys=0.34, 
> real=2.05 secs]
> 2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-sweep: 
> 1441.773/1618.869 secs] [Times: user=2518.60 sys=59.25, real=1618.62 secs]
> 2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-reset-start]
> 2017-03-10T10:15:33.126+0800: 4555923.762: [CMS-concurrent-reset: 0.215/0.215 
> secs] [Times: user=1.23 sys=0.08, real=0.22 secs]
> 2017-03-10T10:15:33.236+0800: 4555923.873: [GC2017-03-10T10:15:33.237+0800: 
> 4555923.873: [ParNew: 3067011K->340736K(3067136K), 2.4140270 secs] 
> 80615855K->78315999K(100322560K), 2.4144230 secs] [Times: user=4.63 sys=0.36, 
> real=2.41 secs]
> 2017-03-10T10:15:35.655+0800: 4555926.292: [GC [1 CMS-initial-mark: 
> 77975263K(97255424K)] 78316286K(100322560K), 0.0149650 secs] [Times: 
> user=0.01 sys=0.00, real=0.01 secs]
> 2017-03-10T10:15:35.671+0800: 4555926.307: [CMS-concurrent-mark-start]
> 2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-mark: 0.427/0.427 
> secs] [Times: user=5.72 sys=0.05, real=0.43 secs]
> 2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-preclean-start]
> 2017-03-10T10:15:36.291+0800: 4555926.928: [CMS-concurrent-preclean: 
> 0.192/0.193 secs] [Times: user=0.80 sys=0.03, real=0.19 secs]
> 2017-03-10T10:15:36.291+0800: 4555926.928: 
> [CMS-concurrent-abortable-preclean-start]
> 2017-03-10T10:15:37.378+0800: 4555928.014: [GC2017-03-10T10:15:37.378+0800: 
> 4555928.014: [ParNew: 3067083K->340736K(3067136K), 2.6221190 secs] 
> 81042347K->78771078K(100322560K), 2.6224970 secs] [Times: user=4.79 sys=0.48, 
> real=2.62 secs]
> 2017-03-10T10:15:41.012+0800: 4555931.648: 
> [CMS-concurrent-abortable-preclean: 2.083/4.721 secs] [Times: user=13.51 
> sys=0.87, real=4.72 secs]
> 2017-03-10T10:15:41.015+0800: 4555931.652: [GC[YG occupancy: 2011637 K 
> (3067136 K)]2017-03-10T10:15:41.016+0800: 4555931.652: 
> [GC2017-03-10T10:15:41.016+0800: 4555931.652: [ParNew: 
> 2011637K->340736K(3067136K), 2.0773980 secs] 
> 80441979K->79117650K(100322560K), 2.0777380 secs] [Times: user=4.09 sys=0.38, 
> real=2.07 secs]
> regionserver  JVM config:
> export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:PermSize=256m 
> -XX:MaxPermSize=256m -Xms96G -Xmx96G"
> export HBASE_OPTS="$HBASE_OPTS -Djava.net.preferIPv4Stack=true
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=60 
> -XX:+CMSParallelRemarkEnabled -XX:+CMSConcurrentMTEnabled
> -XX:ParallelGCThreads=40 -XX:+DisableExplicitGC -XX:+PrintGCDetails 
> -XX:+PrintGCDateStamps -verbose:gc
> -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=1 
> -XX:+CMSScavengeBeforeRemark -XX:+HeapDumpOnOutOfMemoryError



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Moving 2.0 forward

[Stack]

On Wed, Mar 8, 2017 at 4:06 PM, Jerry He  wrote:

> Thanks for the write-up, Stack.
>
> I take the liberty to make the Spark item as:
>
> 4.3 hbase-spark
>  ktczrlKHK8N4SZzs/edit#heading=h.24l014zh4tmp>
>
> 4.3.1 Status=
>  ktczrlKHK8N4SZzs/edit#heading=h.qx14vj8gbs1e>
> IN_PROGRESS
>
> 4.3.2 Owner=
>  ktczrlKHK8N4SZzs/edit#heading=h.h3ncb1tfs5e8>Jerry
> and team
>
> We will see how much we can dot to fix-up and improve.
>
>
I love it Jerry.

What you thinking regards fixup? Talk out loud I'd say because others
interested.

Thanks for edit on doc.

St.Ack



> Thanks.
>
> Jerry
>

Re: Moving 2.0 forward

[Stack]

On Tue, Mar 7, 2017 at 1:51 PM, Josh Elser  wrote:

> Thanks for pulling in the FS Quotas work, Stack. I'm trying to cross the
> last T's and dot the last I's.
>
> The biggest thing I know I need to do still is to write a new chapter to
> the book. After that, I'd start entertaining larger reviews/discussions to
> merge the feature into master. Anyone with free time (giggles) is more than
> welcome to start perusing :)
>
>
Out of interest, this could come in after 2.0 Josh? Any 2.0 specific needs
to make this work?

Meantime, updated the 2.0 doc 1.

Thanks Josh,
St.Ack

1.
https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit#



>
> Stack wrote:
>
>> Just a quick update (Please feel free to amend your own status in our
>> hbase-2.0 doc [1]).
>>
>> On the criticals:
>>
>> + Little progress on the blocker AMv2. A month or more necessary still
>> before code complete (Estimate)
>> + Accordion (inmemory compaction) progressing nicely as is the offheap
>> read/write path. Both are almost done.
>> + Rolling upgrade testing. No work done.
>> + No recent work on core decision tasks (clean-up narrative around RPC
>> timeout, hbase:meta on master or not, batch vs partial semantic, etc.)
>>
>> Non-criticals/Ancillaries
>>
>> + Async client and C++ client are both making good progress. Not done.
>> + Backup/Restore is making good progress
>> + RegionServer-based assignment got a bunch of scrutiny lately and is now
>> 'done'.
>> + FileSystem Quotas making good progress.
>>
>> I'm seeing another month or two at least before branch and probably three.
>> See doc [1] for more detail.
>>
>> Yours,
>> St.Ack
>>
>> 1.  https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
>> ktczrlKHK8N4SZzs/edit#
>>
>>
>>
>>
>>
>> On Sun, Jan 29, 2017 at 9:16 PM, Stack  wrote:
>>
>> On Thu, Jan 26, 2017 at 11:49 PM, ramkrishna vasudevan<
>>> ramkrishna.s.vasude...@gmail.com>  wrote:
>>>
>>> Hi All

 Thanks Stack. The doc looks great. The offheap write path/read path- I
 think from the read path perspective we have some good feedback from
 Alibaba folks.

 Agree.
>>>
>>>
>>> The write path subtasks are all done. We are currently working on some
 perf
 results that would help us to come up with some docs that suggests best
 configs and tunings for the offheap write path configurations.


 Thanks Ram. Would be good to hear what configs you are looking to
>>> implement as default so those of us also starting to test can enable them
>>> to get you feedback.
>>>
>>> Also suggest you fill the above short status into the doc (You are
>>> keeping
>>> up full status elsewhere). I've been trying to add status as I see it
>>> popping up; e.g. Enis did a nice state-of-the-C++ client recently up in
>>> JIRA and I added pointer to the 2.0 doc. Anyone else working on 2.0
>>> features, would be good if you kept a short state in this overview doc;
>>> just ask for edit perms.
>>>
>>> Thanks,
>>> St.Ack
>>>
>>>
>>>
>>>
>>>
>>> Regards
 Ram

 On Thu, Jan 19, 2017 at 5:37 AM, Andrew Purtell help test.
>
>
> On Jan 18, 2017, at 2:53 PM, Stack  wrote:
>>
>> On Wed, Jan 18, 2017 at 2:26 PM, Francis Liu
>>>
>> wrote:

> Hi Stack,
>>> I'd like to get split meta (HBASE-112288) in 2.x as well. I can have
>>>
>> a

> 2.x
>
>> draft up next week. Was working on the 1.x version internally.
>>> Also if you'd like I can be the owner for rsgroups as well.
>>> Thanks,Francis
>>>
>>>
>>>
>>> I added splitting meta as a possible and had you and I as owner on
>>>
>> rsgroups (I was doing to do a bit of testing and doc for this
>>
> feature).

> Would love to see splittable meta show up. Needs to be rolling
>>
> upgradeable
>
>> though. Lets chat up on the issue.
>> St.Ack
>>
>>
>>
>>
>>
>>>
>>> On Wednesday, January 18, 2017 11:29 AM, Stack
>>> wrote:
>>>
>>>
>>> Done Thiruvel (And thanks Guanghao for adding hbase-replication).
>>> St.Ack
>>>
>>> On Tue, Jan 17, 2017 at 6:11 PM, Thiruvel Thirumoolan<
>>> thiru...@yahoo-inc.com.invalid>  wrote:
>>>
>>> Hi Stack,
 I would like to add Favored Nodes to the ancillary section.
 HBASE-15531: Favored Nodes EnhancementsStatus: Active

>>> development.Owner:
>
>> Thiruvel Thanks!-Thiruvel

On Monday, January 16, 2017 2:10 PM, Stack

>>> wrote:

>
 On Mon, Jan 16, 2017 at 3:01 AM, Guanghao Zhang

[jira] [Created] (HBASE-17767) hbase CMS gc pause program

[gehaijiang (JIRA)]

gehaijiang created HBASE-17767:
--

 Summary: hbase CMS  gc  pause program
 Key: HBASE-17767
 URL: https://issues.apache.org/jira/browse/HBASE-17767
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.1.2
Reporter: gehaijiang


cms gc pause serious program:

2017-03-10T10:15:29.524+0800: 4555920.160: [GC2017-03-10T10:15:29.524+0800: 
4555920.160: [ParNew: 3067133K->340736K(3067136K), 2.0586980 secs] 
80328431K->78058138K(100322560K), 2.0590280 secs] [Times: user=3.94 sys=0.34, 
real=2.05 secs]
2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-sweep: 
1441.773/1618.869 secs] [Times: user=2518.60 sys=59.25, real=1618.62 secs]
2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-reset-start]
2017-03-10T10:15:33.126+0800: 4555923.762: [CMS-concurrent-reset: 0.215/0.215 
secs] [Times: user=1.23 sys=0.08, real=0.22 secs]
2017-03-10T10:15:33.236+0800: 4555923.873: [GC2017-03-10T10:15:33.237+0800: 
4555923.873: [ParNew: 3067011K->340736K(3067136K), 2.4140270 secs] 
80615855K->78315999K(100322560K), 2.4144230 secs] [Times: user=4.63 sys=0.36, 
real=2.41 secs]
2017-03-10T10:15:35.655+0800: 4555926.292: [GC [1 CMS-initial-mark: 
77975263K(97255424K)] 78316286K(100322560K), 0.0149650 secs] [Times: user=0.01 
sys=0.00, real=0.01 secs]
2017-03-10T10:15:35.671+0800: 4555926.307: [CMS-concurrent-mark-start]
2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-mark: 0.427/0.427 
secs] [Times: user=5.72 sys=0.05, real=0.43 secs]
2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-preclean-start]
2017-03-10T10:15:36.291+0800: 4555926.928: [CMS-concurrent-preclean: 
0.192/0.193 secs] [Times: user=0.80 sys=0.03, real=0.19 secs]
2017-03-10T10:15:36.291+0800: 4555926.928: 
[CMS-concurrent-abortable-preclean-start]
2017-03-10T10:15:37.378+0800: 4555928.014: [GC2017-03-10T10:15:37.378+0800: 
4555928.014: [ParNew: 3067083K->340736K(3067136K), 2.6221190 secs] 
81042347K->78771078K(100322560K), 2.6224970 secs] [Times: user=4.79 sys=0.48, 
real=2.62 secs]
2017-03-10T10:15:41.012+0800: 4555931.648: [CMS-concurrent-abortable-preclean: 
2.083/4.721 secs] [Times: user=13.51 sys=0.87, real=4.72 secs]
2017-03-10T10:15:41.015+0800: 4555931.652: [GC[YG occupancy: 2011637 K (3067136 
K)]2017-03-10T10:15:41.016+0800: 4555931.652: [GC2017-03-10T10:15:41.016+0800: 
4555931.652: [ParNew: 2011637K->340736K(3067136K), 2.0773980 secs] 
80441979K->79117650K(100322560K), 2.0777380 secs] [Times: user=4.09 sys=0.38, 
real=2.07 secs]






regionserver  JVM config:

export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:PermSize=256m 
-XX:MaxPermSize=256m -Xms96G -Xmx96G"
export HBASE_OPTS="$HBASE_OPTS -Djava.net.preferIPv4Stack=true
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=60 
-XX:+CMSParallelRemarkEnabled -XX:+CMSConcurrentMTEnabled
-XX:ParallelGCThreads=40 -XX:+DisableExplicitGC -XX:+PrintGCDetails 
-XX:+PrintGCDateStamps -verbose:gc
-XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=1 
-XX:+CMSScavengeBeforeRemark -XX:+HeapDumpOnOutOfMemoryError





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: A suggestion for releasing major versions faster (Was: NOTICE: Nice testimony on benefits of the offheap read-path work now up on our blog)

[Phil Yang]

> To rephrase myself - I think we shouldn't be taking stance that
> MajorBranchX can't be released until features f1, f2, f2,...fN are all
completed.

Agree, we should not be blocked by any feature.

Some new features don't conflict with current codes, I think we can just
commit to branches and
add a how-to-enable document into HBaseBook/ReleaseNote or enable it by
default, when it is ready.

For the issues that may break the branch to can-not-release state, I think
we should never get them in and open a feature branch,
merge them when they are all ready.

Thanks,
Phil


2017-03-10 9:41 GMT+08:00 Mikhail Antonov :

> To rephrase myself - I think we shouldn't be taking stance that
> MajorBranchX can't be released until features f1, f2, f2,...fN are all
> completed.
>
> That's just not going to fly. There will always be features suddenly way
> more complicated than they appeared, features that don't have enough
> manpower behind them, features that are found to be less important then
> they were thought to be.
>
> Instead we probably should say "branch-2 is being cut; (or, in kernel
> parlance, here's merge window dates) whatever didn't make their way into it
> will have a chance to go to branch-2.1, branch-2.2, branch-3".
>
>
>
> On Thu, Mar 9, 2017 at 5:29 PM, Mikhail Antonov 
> wrote:
>
> > >>"The only way forward for saving 2.0 at this point is to *make the
> > branch
> > and spin the RC."
> >
> > big +1 to that.
> >
> > I do also think that talking about planning in the context of open-source
> > community development is a bit hard - there's no planning but what people
> > do. If we release regularly, however, we would naturally be picking
> > important things (features, fixes, optimizations), because things that
> were
> > planned but not completed up to some usable checkpoint state were, by the
> > very definition, not truly important enough in the grand schema of
> things.
> > From the perspective, releasing regularly is more important than
> preparing
> > a list of must-have things and waiting to cross the last work item out
> > before the release can be made.
> >
> > That does pose a problem of partially-completed features. Two viable ways
> > to deal with them that I see are a) use feature branches more and b)
> don't
> > release off master (or, commit new code to some 'unstable' branch first,
> > then pull to master).
> >
> > Thoughts?
> >
> > -Mikhail
> >
> > On Thu, Mar 9, 2017 at 1:57 PM, Andrew Purtell 
> > wrote:
> >
> >> > For 2.0 release or future major release, what we need is planning -
> >> THEME
> >> ​> ​
> >> (what is the biggest excitement for the release) and MUST-HAVE FEATURES.
> >> ​> ​
> >> All must-have features should have an owner and some estimate of
> >> completing
> >> ​> ​
> >> time.
> >>
> >> ​With all due respect, this is just talk. Appeals to planning and "must
> >> have" features has an import that decays proportionally to the time
> since
> >> the last time we had some words about it. 2.0 keeps slipping, slipping,
> >> slipping. The PMC needs to come to terms with the actual amount of
> >> development bandwidth we have available and set a cut point. Make it
> >> happen, "do-ocracy" style. ​
> >>
> >>
> >>
> >> On Thu, Mar 9, 2017 at 1:52 PM, Andrew Purtell 
> >> wrote:
> >>
> >> > Oh, I don't know. I may never be comfortable with a backport of MOB
> into
> >> > branch-1, but a branch-2 including it would be fine.
> >> >
> >> > And my point is not that making a branch-2 out of branch-1 is
> desirable,
> >> > simply that it could be the most practical way forward if we are stuck
> >> on
> >> > master with too much unfinished work that cannot be reverted in order
> to
> >> > make a production ready release.
> >> >
> >> >
> >> > On Thu, Mar 9, 2017 at 1:12 PM, Stephen Jiang <
> syuanjiang...@gmail.com>
> >> > wrote:
> >> >
> >> >> I don't see a point to have branch-2 from branch-1.  For
> >> customer/users,
> >> >> we
> >> >> always can have a 1.x release to give them all the features they want
> >> from
> >> >> branch-1.
> >> >>
> >> >> My understand is that one of the difference of major release and
> minor
> >> >> release is that major release could break some backward
> >> compatibility.  If
> >> >> any features that in master, but not in branch-1, as long as not
> >> breaking
> >> >> backward compatible, the owner of the feature always can back-port to
> >> >> branch-1 if they desire.  Today we don't have voting process to block
> >> >> that.
> >> >>
> >> >> For 2.0 release or future major release, what we need is planning -
> >> THEME
> >> >> (what is the biggest excitement for the release) and MUST-HAVE
> >> FEATURES.
> >> >> All must-have features should have an owner and some estimate of
> >> >> completing
> >> >> time.  Once the consensus is reached, then next step is execution -
> the
> >> >> release time would be based on progress of these must-have features.
> >> >>
> >> >> 

[jira] [Resolved] (HBASE-11033) Allow test-patch.sh to run selected test(s) if patch contains changes to test classes only

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-11033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HBASE-11033.

Resolution: Won't Fix

> Allow test-patch.sh to run selected test(s) if patch contains changes to test 
> classes only
> --
>
> Key: HBASE-11033
> URL: https://issues.apache.org/jira/browse/HBASE-11033
> Project: HBase
>  Issue Type: Task
>Reporter: Ted Yu
>Priority: Minor
>  Labels: precommit
>
> A patch may touch test classes only.
> In this case, test-patch.sh can detect the tests affected and run these tests 
> only.
> Special case: if TestingUtility (HConnectionTestingUtility, 
> HBaseTestingUtility, etc) is changed, all tests need to be run.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: A suggestion for releasing major versions faster (Was: NOTICE: Nice testimony on benefits of the offheap read-path work now up on our blog)

[Mikhail Antonov]

To rephrase myself - I think we shouldn't be taking stance that
MajorBranchX can't be released until features f1, f2, f2,...fN are all
completed.

That's just not going to fly. There will always be features suddenly way
more complicated than they appeared, features that don't have enough
manpower behind them, features that are found to be less important then
they were thought to be.

Instead we probably should say "branch-2 is being cut; (or, in kernel
parlance, here's merge window dates) whatever didn't make their way into it
will have a chance to go to branch-2.1, branch-2.2, branch-3".



On Thu, Mar 9, 2017 at 5:29 PM, Mikhail Antonov  wrote:

> >>"The only way forward for saving 2.0 at this point is to *make the
> branch
> and spin the RC."
>
> big +1 to that.
>
> I do also think that talking about planning in the context of open-source
> community development is a bit hard - there's no planning but what people
> do. If we release regularly, however, we would naturally be picking
> important things (features, fixes, optimizations), because things that were
> planned but not completed up to some usable checkpoint state were, by the
> very definition, not truly important enough in the grand schema of things.
> From the perspective, releasing regularly is more important than preparing
> a list of must-have things and waiting to cross the last work item out
> before the release can be made.
>
> That does pose a problem of partially-completed features. Two viable ways
> to deal with them that I see are a) use feature branches more and b) don't
> release off master (or, commit new code to some 'unstable' branch first,
> then pull to master).
>
> Thoughts?
>
> -Mikhail
>
> On Thu, Mar 9, 2017 at 1:57 PM, Andrew Purtell 
> wrote:
>
>> > For 2.0 release or future major release, what we need is planning -
>> THEME
>> ​> ​
>> (what is the biggest excitement for the release) and MUST-HAVE FEATURES.
>> ​> ​
>> All must-have features should have an owner and some estimate of
>> completing
>> ​> ​
>> time.
>>
>> ​With all due respect, this is just talk. Appeals to planning and "must
>> have" features has an import that decays proportionally to the time since
>> the last time we had some words about it. 2.0 keeps slipping, slipping,
>> slipping. The PMC needs to come to terms with the actual amount of
>> development bandwidth we have available and set a cut point. Make it
>> happen, "do-ocracy" style. ​
>>
>>
>>
>> On Thu, Mar 9, 2017 at 1:52 PM, Andrew Purtell 
>> wrote:
>>
>> > Oh, I don't know. I may never be comfortable with a backport of MOB into
>> > branch-1, but a branch-2 including it would be fine.
>> >
>> > And my point is not that making a branch-2 out of branch-1 is desirable,
>> > simply that it could be the most practical way forward if we are stuck
>> on
>> > master with too much unfinished work that cannot be reverted in order to
>> > make a production ready release.
>> >
>> >
>> > On Thu, Mar 9, 2017 at 1:12 PM, Stephen Jiang 
>> > wrote:
>> >
>> >> I don't see a point to have branch-2 from branch-1.  For
>> customer/users,
>> >> we
>> >> always can have a 1.x release to give them all the features they want
>> from
>> >> branch-1.
>> >>
>> >> My understand is that one of the difference of major release and minor
>> >> release is that major release could break some backward
>> compatibility.  If
>> >> any features that in master, but not in branch-1, as long as not
>> breaking
>> >> backward compatible, the owner of the feature always can back-port to
>> >> branch-1 if they desire.  Today we don't have voting process to block
>> >> that.
>> >>
>> >> For 2.0 release or future major release, what we need is planning -
>> THEME
>> >> (what is the biggest excitement for the release) and MUST-HAVE
>> FEATURES.
>> >> All must-have features should have an owner and some estimate of
>> >> completing
>> >> time.  Once the consensus is reached, then next step is execution - the
>> >> release time would be based on progress of these must-have features.
>> >>
>> >> Thanks
>> >> Stephen
>> >>
>> >> On Thu, Mar 9, 2017 at 11:53 AM, Ashu Pachauri 
>> >> wrote:
>> >>
>> >> > In my opinion, a major release is based on two simultaneous
>> decisions:
>> >> >
>> >> > 1. Is it time; may be a year is a good time frame? (It's useless
>> >> > accumulating too much code that is not battle tested and then expect
>> >> people
>> >> > to deploy it to production without experiencing a plethora of
>> issues.)
>> >> >
>> >> > 2. Is there at least one "major feature" that is complete ?
>> >> >
>> >> > I think if we can answer yes to both these questions at any point in
>> >> time,
>> >> > it's a good idea to cut the RC and ask people to start testing it.
>> >> >
>> >> > the only way forward for saving 2.0 at this point is to *make the
>> branch
>> >> > and
>> >> > > spin the RC
>> >> >
>> >> > +1
>> >> >
>> >> >
>> >> > On 

Re: A suggestion for releasing major versions faster (Was: NOTICE: Nice testimony on benefits of the offheap read-path work now up on our blog)

[Duo Zhang]

Let's just cut branch-2 ASAP and set a deadline for the partially-completed
features?

2017-03-10 9:29 GMT+08:00 Mikhail Antonov :

> >>"The only way forward for saving 2.0 at this point is to *make the branch
> and spin the RC."
>
> big +1 to that.
>
> I do also think that talking about planning in the context of open-source
> community development is a bit hard - there's no planning but what people
> do. If we release regularly, however, we would naturally be picking
> important things (features, fixes, optimizations), because things that were
> planned but not completed up to some usable checkpoint state were, by the
> very definition, not truly important enough in the grand schema of things.
> From the perspective, releasing regularly is more important than preparing
> a list of must-have things and waiting to cross the last work item out
> before the release can be made.
>
> That does pose a problem of partially-completed features. Two viable ways
> to deal with them that I see are a) use feature branches more and b) don't
> release off master (or, commit new code to some 'unstable' branch first,
> then pull to master).
>
> Thoughts?
>
> -Mikhail
>
> On Thu, Mar 9, 2017 at 1:57 PM, Andrew Purtell 
> wrote:
>
> > > For 2.0 release or future major release, what we need is planning -
> THEME
> > ​> ​
> > (what is the biggest excitement for the release) and MUST-HAVE FEATURES.
> > ​> ​
> > All must-have features should have an owner and some estimate of
> completing
> > ​> ​
> > time.
> >
> > ​With all due respect, this is just talk. Appeals to planning and "must
> > have" features has an import that decays proportionally to the time since
> > the last time we had some words about it. 2.0 keeps slipping, slipping,
> > slipping. The PMC needs to come to terms with the actual amount of
> > development bandwidth we have available and set a cut point. Make it
> > happen, "do-ocracy" style. ​
> >
> >
> >
> > On Thu, Mar 9, 2017 at 1:52 PM, Andrew Purtell 
> > wrote:
> >
> > > Oh, I don't know. I may never be comfortable with a backport of MOB
> into
> > > branch-1, but a branch-2 including it would be fine.
> > >
> > > And my point is not that making a branch-2 out of branch-1 is
> desirable,
> > > simply that it could be the most practical way forward if we are stuck
> on
> > > master with too much unfinished work that cannot be reverted in order
> to
> > > make a production ready release.
> > >
> > >
> > > On Thu, Mar 9, 2017 at 1:12 PM, Stephen Jiang  >
> > > wrote:
> > >
> > >> I don't see a point to have branch-2 from branch-1.  For
> customer/users,
> > >> we
> > >> always can have a 1.x release to give them all the features they want
> > from
> > >> branch-1.
> > >>
> > >> My understand is that one of the difference of major release and minor
> > >> release is that major release could break some backward compatibility.
> > If
> > >> any features that in master, but not in branch-1, as long as not
> > breaking
> > >> backward compatible, the owner of the feature always can back-port to
> > >> branch-1 if they desire.  Today we don't have voting process to block
> > >> that.
> > >>
> > >> For 2.0 release or future major release, what we need is planning -
> > THEME
> > >> (what is the biggest excitement for the release) and MUST-HAVE
> FEATURES.
> > >> All must-have features should have an owner and some estimate of
> > >> completing
> > >> time.  Once the consensus is reached, then next step is execution -
> the
> > >> release time would be based on progress of these must-have features.
> > >>
> > >> Thanks
> > >> Stephen
> > >>
> > >> On Thu, Mar 9, 2017 at 11:53 AM, Ashu Pachauri 
> > >> wrote:
> > >>
> > >> > In my opinion, a major release is based on two simultaneous
> decisions:
> > >> >
> > >> > 1. Is it time; may be a year is a good time frame? (It's useless
> > >> > accumulating too much code that is not battle tested and then expect
> > >> people
> > >> > to deploy it to production without experiencing a plethora of
> issues.)
> > >> >
> > >> > 2. Is there at least one "major feature" that is complete ?
> > >> >
> > >> > I think if we can answer yes to both these questions at any point in
> > >> time,
> > >> > it's a good idea to cut the RC and ask people to start testing it.
> > >> >
> > >> > the only way forward for saving 2.0 at this point is to *make the
> > branch
> > >> > and
> > >> > > spin the RC
> > >> >
> > >> > +1
> > >> >
> > >> >
> > >> > On Thu, Mar 9, 2017 at 11:25 AM, Andrew Purtell <
> apurt...@apache.org>
> > >> > wrote:
> > >> >
> > >> > > The only way forward for saving 2.0 at this point is to *make the
> > >> branch
> > >> > > and spin the RC. *Just do it. Kick out by revert what obviously
> > isn't
> > >> > > ready. Solicit help in getting partially finished things into
> > working
> > >> > > state. Kick them out too if the help does not arrive.
> > >> > >
> > >> > 

Re: A suggestion for releasing major versions faster (Was: NOTICE: Nice testimony on benefits of the offheap read-path work now up on our blog)

[Mikhail Antonov]

>>"The only way forward for saving 2.0 at this point is to *make the branch
and spin the RC."

big +1 to that.

I do also think that talking about planning in the context of open-source
community development is a bit hard - there's no planning but what people
do. If we release regularly, however, we would naturally be picking
important things (features, fixes, optimizations), because things that were
planned but not completed up to some usable checkpoint state were, by the
very definition, not truly important enough in the grand schema of things.
>From the perspective, releasing regularly is more important than preparing
a list of must-have things and waiting to cross the last work item out
before the release can be made.

That does pose a problem of partially-completed features. Two viable ways
to deal with them that I see are a) use feature branches more and b) don't
release off master (or, commit new code to some 'unstable' branch first,
then pull to master).

Thoughts?

-Mikhail

On Thu, Mar 9, 2017 at 1:57 PM, Andrew Purtell  wrote:

> > For 2.0 release or future major release, what we need is planning - THEME
> ​> ​
> (what is the biggest excitement for the release) and MUST-HAVE FEATURES.
> ​> ​
> All must-have features should have an owner and some estimate of completing
> ​> ​
> time.
>
> ​With all due respect, this is just talk. Appeals to planning and "must
> have" features has an import that decays proportionally to the time since
> the last time we had some words about it. 2.0 keeps slipping, slipping,
> slipping. The PMC needs to come to terms with the actual amount of
> development bandwidth we have available and set a cut point. Make it
> happen, "do-ocracy" style. ​
>
>
>
> On Thu, Mar 9, 2017 at 1:52 PM, Andrew Purtell 
> wrote:
>
> > Oh, I don't know. I may never be comfortable with a backport of MOB into
> > branch-1, but a branch-2 including it would be fine.
> >
> > And my point is not that making a branch-2 out of branch-1 is desirable,
> > simply that it could be the most practical way forward if we are stuck on
> > master with too much unfinished work that cannot be reverted in order to
> > make a production ready release.
> >
> >
> > On Thu, Mar 9, 2017 at 1:12 PM, Stephen Jiang 
> > wrote:
> >
> >> I don't see a point to have branch-2 from branch-1.  For customer/users,
> >> we
> >> always can have a 1.x release to give them all the features they want
> from
> >> branch-1.
> >>
> >> My understand is that one of the difference of major release and minor
> >> release is that major release could break some backward compatibility.
> If
> >> any features that in master, but not in branch-1, as long as not
> breaking
> >> backward compatible, the owner of the feature always can back-port to
> >> branch-1 if they desire.  Today we don't have voting process to block
> >> that.
> >>
> >> For 2.0 release or future major release, what we need is planning -
> THEME
> >> (what is the biggest excitement for the release) and MUST-HAVE FEATURES.
> >> All must-have features should have an owner and some estimate of
> >> completing
> >> time.  Once the consensus is reached, then next step is execution - the
> >> release time would be based on progress of these must-have features.
> >>
> >> Thanks
> >> Stephen
> >>
> >> On Thu, Mar 9, 2017 at 11:53 AM, Ashu Pachauri 
> >> wrote:
> >>
> >> > In my opinion, a major release is based on two simultaneous decisions:
> >> >
> >> > 1. Is it time; may be a year is a good time frame? (It's useless
> >> > accumulating too much code that is not battle tested and then expect
> >> people
> >> > to deploy it to production without experiencing a plethora of issues.)
> >> >
> >> > 2. Is there at least one "major feature" that is complete ?
> >> >
> >> > I think if we can answer yes to both these questions at any point in
> >> time,
> >> > it's a good idea to cut the RC and ask people to start testing it.
> >> >
> >> > the only way forward for saving 2.0 at this point is to *make the
> branch
> >> > and
> >> > > spin the RC
> >> >
> >> > +1
> >> >
> >> >
> >> > On Thu, Mar 9, 2017 at 11:25 AM, Andrew Purtell 
> >> > wrote:
> >> >
> >> > > The only way forward for saving 2.0 at this point is to *make the
> >> branch
> >> > > and spin the RC. *Just do it. Kick out by revert what obviously
> isn't
> >> > > ready. Solicit help in getting partially finished things into
> working
> >> > > state. Kick them out too if the help does not arrive.
> >> > >
> >> > > Maybe too much is in a half done state and the scale of effort for
> >> those
> >> > > reverts is too high. Fine. Renumber master as 3.0, and make a
> branch-2
> >> > from
> >> > > branch-1 and backport a select number of things from master into the
> >> new
> >> > > branch-2. Then release. I do a variation of this for the $dayjob so
> >> would
> >> > > be your man to turn to for driving this if that's the 

branch-2

[Ted Yu]

Hi,
I was wondering why there is branch-2 ?

https://github.com/apache/hbase/tree/branch-2

As far as I know, there is no consensus on when branch-2 should be created.

Thanks

Re: A suggestion for releasing major versions faster (Was: NOTICE: Nice testimony on benefits of the offheap read-path work now up on our blog)

[Andrew Purtell]

> For 2.0 release or future major release, what we need is planning - THEME
​> ​
(what is the biggest excitement for the release) and MUST-HAVE FEATURES.
​> ​
All must-have features should have an owner and some estimate of completing
​> ​
time.

​With all due respect, this is just talk. Appeals to planning and "must
have" features has an import that decays proportionally to the time since
the last time we had some words about it. 2.0 keeps slipping, slipping,
slipping. The PMC needs to come to terms with the actual amount of
development bandwidth we have available and set a cut point. Make it
happen, "do-ocracy" style. ​



On Thu, Mar 9, 2017 at 1:52 PM, Andrew Purtell  wrote:

> Oh, I don't know. I may never be comfortable with a backport of MOB into
> branch-1, but a branch-2 including it would be fine.
>
> And my point is not that making a branch-2 out of branch-1 is desirable,
> simply that it could be the most practical way forward if we are stuck on
> master with too much unfinished work that cannot be reverted in order to
> make a production ready release.
>
>
> On Thu, Mar 9, 2017 at 1:12 PM, Stephen Jiang 
> wrote:
>
>> I don't see a point to have branch-2 from branch-1.  For customer/users,
>> we
>> always can have a 1.x release to give them all the features they want from
>> branch-1.
>>
>> My understand is that one of the difference of major release and minor
>> release is that major release could break some backward compatibility.  If
>> any features that in master, but not in branch-1, as long as not breaking
>> backward compatible, the owner of the feature always can back-port to
>> branch-1 if they desire.  Today we don't have voting process to block
>> that.
>>
>> For 2.0 release or future major release, what we need is planning - THEME
>> (what is the biggest excitement for the release) and MUST-HAVE FEATURES.
>> All must-have features should have an owner and some estimate of
>> completing
>> time.  Once the consensus is reached, then next step is execution - the
>> release time would be based on progress of these must-have features.
>>
>> Thanks
>> Stephen
>>
>> On Thu, Mar 9, 2017 at 11:53 AM, Ashu Pachauri 
>> wrote:
>>
>> > In my opinion, a major release is based on two simultaneous decisions:
>> >
>> > 1. Is it time; may be a year is a good time frame? (It's useless
>> > accumulating too much code that is not battle tested and then expect
>> people
>> > to deploy it to production without experiencing a plethora of issues.)
>> >
>> > 2. Is there at least one "major feature" that is complete ?
>> >
>> > I think if we can answer yes to both these questions at any point in
>> time,
>> > it's a good idea to cut the RC and ask people to start testing it.
>> >
>> > the only way forward for saving 2.0 at this point is to *make the branch
>> > and
>> > > spin the RC
>> >
>> > +1
>> >
>> >
>> > On Thu, Mar 9, 2017 at 11:25 AM, Andrew Purtell 
>> > wrote:
>> >
>> > > The only way forward for saving 2.0 at this point is to *make the
>> branch
>> > > and spin the RC. *Just do it. Kick out by revert what obviously isn't
>> > > ready. Solicit help in getting partially finished things into working
>> > > state. Kick them out too if the help does not arrive.
>> > >
>> > > Maybe too much is in a half done state and the scale of effort for
>> those
>> > > reverts is too high. Fine. Renumber master as 3.0, and make a branch-2
>> > from
>> > > branch-1 and backport a select number of things from master into the
>> new
>> > > branch-2. Then release. I do a variation of this for the $dayjob so
>> would
>> > > be your man to turn to for driving this if that's the way forward.
>> > >
>> > > I know it's easy to recommend the labor of others. Depending on what
>> we
>> > are
>> > > going to do I can talk to work about freeing up time to help.
>> > >
>> > >
>> > > On Thu, Mar 9, 2017 at 11:18 AM, Stack  wrote:
>> > >
>> > > > On Thu, Mar 9, 2017 at 12:34 AM, Phil Yang 
>> > > wrote:
>> > > > >
>> > > > >
>> > > > > So my suggestion is cutting branch-x faster and have some fixed
>> > period,
>> > > > for
>> > > > > example, six month or one year?
>> > > > >
>> > > >
>> > > >
>> > > > You are right Phil.
>> > > >
>> > > > The Release Managers for the minor releases have been doing a good
>> job
>> > > > keeping up a decent release cadence but we have an abysmal track
>> record
>> > > > when it comes to pushing out majors. First we were afraid to commit
>> --
>> > > > witness how long it took us to get to a 1.0 -- and then pushing out
>> the
>> > > 1.0
>> > > > took a monster effort. 2.0 looks to be a repeat of the errors of
>> 1.0.
>> > My
>> > > > sense is that 2.0 is beyond saving at this stage.
>> > > >
>> > > > Can we do 3.0 different? As per your suggestion?
>> > > >
>> > > > St.Ack
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Best regards,
>> > >
>> > >- Andy
>> > >
>> 

Re: A suggestion for releasing major versions faster (Was: NOTICE: Nice testimony on benefits of the offheap read-path work now up on our blog)

[Andrew Purtell]

Oh, I don't know. I may never be comfortable with a backport of MOB into
branch-1, but a branch-2 including it would be fine.

And my point is not that making a branch-2 out of branch-1 is desirable,
simply that it could be the most practical way forward if we are stuck on
master with too much unfinished work that cannot be reverted in order to
make a production ready release.


On Thu, Mar 9, 2017 at 1:12 PM, Stephen Jiang 
wrote:

> I don't see a point to have branch-2 from branch-1.  For customer/users, we
> always can have a 1.x release to give them all the features they want from
> branch-1.
>
> My understand is that one of the difference of major release and minor
> release is that major release could break some backward compatibility.  If
> any features that in master, but not in branch-1, as long as not breaking
> backward compatible, the owner of the feature always can back-port to
> branch-1 if they desire.  Today we don't have voting process to block that.
>
> For 2.0 release or future major release, what we need is planning - THEME
> (what is the biggest excitement for the release) and MUST-HAVE FEATURES.
> All must-have features should have an owner and some estimate of completing
> time.  Once the consensus is reached, then next step is execution - the
> release time would be based on progress of these must-have features.
>
> Thanks
> Stephen
>
> On Thu, Mar 9, 2017 at 11:53 AM, Ashu Pachauri 
> wrote:
>
> > In my opinion, a major release is based on two simultaneous decisions:
> >
> > 1. Is it time; may be a year is a good time frame? (It's useless
> > accumulating too much code that is not battle tested and then expect
> people
> > to deploy it to production without experiencing a plethora of issues.)
> >
> > 2. Is there at least one "major feature" that is complete ?
> >
> > I think if we can answer yes to both these questions at any point in
> time,
> > it's a good idea to cut the RC and ask people to start testing it.
> >
> > the only way forward for saving 2.0 at this point is to *make the branch
> > and
> > > spin the RC
> >
> > +1
> >
> >
> > On Thu, Mar 9, 2017 at 11:25 AM, Andrew Purtell 
> > wrote:
> >
> > > The only way forward for saving 2.0 at this point is to *make the
> branch
> > > and spin the RC. *Just do it. Kick out by revert what obviously isn't
> > > ready. Solicit help in getting partially finished things into working
> > > state. Kick them out too if the help does not arrive.
> > >
> > > Maybe too much is in a half done state and the scale of effort for
> those
> > > reverts is too high. Fine. Renumber master as 3.0, and make a branch-2
> > from
> > > branch-1 and backport a select number of things from master into the
> new
> > > branch-2. Then release. I do a variation of this for the $dayjob so
> would
> > > be your man to turn to for driving this if that's the way forward.
> > >
> > > I know it's easy to recommend the labor of others. Depending on what we
> > are
> > > going to do I can talk to work about freeing up time to help.
> > >
> > >
> > > On Thu, Mar 9, 2017 at 11:18 AM, Stack  wrote:
> > >
> > > > On Thu, Mar 9, 2017 at 12:34 AM, Phil Yang 
> > > wrote:
> > > > >
> > > > >
> > > > > So my suggestion is cutting branch-x faster and have some fixed
> > period,
> > > > for
> > > > > example, six month or one year?
> > > > >
> > > >
> > > >
> > > > You are right Phil.
> > > >
> > > > The Release Managers for the minor releases have been doing a good
> job
> > > > keeping up a decent release cadence but we have an abysmal track
> record
> > > > when it comes to pushing out majors. First we were afraid to commit
> --
> > > > witness how long it took us to get to a 1.0 -- and then pushing out
> the
> > > 1.0
> > > > took a monster effort. 2.0 looks to be a repeat of the errors of 1.0.
> > My
> > > > sense is that 2.0 is beyond saving at this stage.
> > > >
> > > > Can we do 3.0 different? As per your suggestion?
> > > >
> > > > St.Ack
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > >- Andy
> > >
> > > If you are given a choice, you believe you have acted freely. - Raymond
> > > Teller (via Peter Watts)
> > >
> >
> >
> >
> > --
> > Thanks and Regards,
> > Ashu Pachauri
> >
>



-- 
Best regards,

   - Andy

If you are given a choice, you believe you have acted freely. - Raymond
Teller (via Peter Watts)

Re: A suggestion for releasing major versions faster (Was: NOTICE: Nice testimony on benefits of the offheap read-path work now up on our blog)

[Stephen Jiang]

I don't see a point to have branch-2 from branch-1.  For customer/users, we
always can have a 1.x release to give them all the features they want from
branch-1.

My understand is that one of the difference of major release and minor
release is that major release could break some backward compatibility.  If
any features that in master, but not in branch-1, as long as not breaking
backward compatible, the owner of the feature always can back-port to
branch-1 if they desire.  Today we don't have voting process to block that.

For 2.0 release or future major release, what we need is planning - THEME
(what is the biggest excitement for the release) and MUST-HAVE FEATURES.
All must-have features should have an owner and some estimate of completing
time.  Once the consensus is reached, then next step is execution - the
release time would be based on progress of these must-have features.

Thanks
Stephen

On Thu, Mar 9, 2017 at 11:53 AM, Ashu Pachauri  wrote:

> In my opinion, a major release is based on two simultaneous decisions:
>
> 1. Is it time; may be a year is a good time frame? (It's useless
> accumulating too much code that is not battle tested and then expect people
> to deploy it to production without experiencing a plethora of issues.)
>
> 2. Is there at least one "major feature" that is complete ?
>
> I think if we can answer yes to both these questions at any point in time,
> it's a good idea to cut the RC and ask people to start testing it.
>
> the only way forward for saving 2.0 at this point is to *make the branch
> and
> > spin the RC
>
> +1
>
>
> On Thu, Mar 9, 2017 at 11:25 AM, Andrew Purtell 
> wrote:
>
> > The only way forward for saving 2.0 at this point is to *make the branch
> > and spin the RC. *Just do it. Kick out by revert what obviously isn't
> > ready. Solicit help in getting partially finished things into working
> > state. Kick them out too if the help does not arrive.
> >
> > Maybe too much is in a half done state and the scale of effort for those
> > reverts is too high. Fine. Renumber master as 3.0, and make a branch-2
> from
> > branch-1 and backport a select number of things from master into the new
> > branch-2. Then release. I do a variation of this for the $dayjob so would
> > be your man to turn to for driving this if that's the way forward.
> >
> > I know it's easy to recommend the labor of others. Depending on what we
> are
> > going to do I can talk to work about freeing up time to help.
> >
> >
> > On Thu, Mar 9, 2017 at 11:18 AM, Stack  wrote:
> >
> > > On Thu, Mar 9, 2017 at 12:34 AM, Phil Yang 
> > wrote:
> > > >
> > > >
> > > > So my suggestion is cutting branch-x faster and have some fixed
> period,
> > > for
> > > > example, six month or one year?
> > > >
> > >
> > >
> > > You are right Phil.
> > >
> > > The Release Managers for the minor releases have been doing a good job
> > > keeping up a decent release cadence but we have an abysmal track record
> > > when it comes to pushing out majors. First we were afraid to commit --
> > > witness how long it took us to get to a 1.0 -- and then pushing out the
> > 1.0
> > > took a monster effort. 2.0 looks to be a repeat of the errors of 1.0.
> My
> > > sense is that 2.0 is beyond saving at this stage.
> > >
> > > Can we do 3.0 different? As per your suggestion?
> > >
> > > St.Ack
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> >- Andy
> >
> > If you are given a choice, you believe you have acted freely. - Raymond
> > Teller (via Peter Watts)
> >
>
>
>
> --
> Thanks and Regards,
> Ashu Pachauri
>

Re: A suggestion for releasing major versions faster (Was: NOTICE: Nice testimony on benefits of the offheap read-path work now up on our blog)

[Ashu Pachauri]

In my opinion, a major release is based on two simultaneous decisions:

1. Is it time; may be a year is a good time frame? (It's useless
accumulating too much code that is not battle tested and then expect people
to deploy it to production without experiencing a plethora of issues.)

2. Is there at least one "major feature" that is complete ?

I think if we can answer yes to both these questions at any point in time,
it's a good idea to cut the RC and ask people to start testing it.

the only way forward for saving 2.0 at this point is to *make the branch and
> spin the RC

+1


On Thu, Mar 9, 2017 at 11:25 AM, Andrew Purtell  wrote:

> The only way forward for saving 2.0 at this point is to *make the branch
> and spin the RC. *Just do it. Kick out by revert what obviously isn't
> ready. Solicit help in getting partially finished things into working
> state. Kick them out too if the help does not arrive.
>
> Maybe too much is in a half done state and the scale of effort for those
> reverts is too high. Fine. Renumber master as 3.0, and make a branch-2 from
> branch-1 and backport a select number of things from master into the new
> branch-2. Then release. I do a variation of this for the $dayjob so would
> be your man to turn to for driving this if that's the way forward.
>
> I know it's easy to recommend the labor of others. Depending on what we are
> going to do I can talk to work about freeing up time to help.
>
>
> On Thu, Mar 9, 2017 at 11:18 AM, Stack  wrote:
>
> > On Thu, Mar 9, 2017 at 12:34 AM, Phil Yang 
> wrote:
> > >
> > >
> > > So my suggestion is cutting branch-x faster and have some fixed period,
> > for
> > > example, six month or one year?
> > >
> >
> >
> > You are right Phil.
> >
> > The Release Managers for the minor releases have been doing a good job
> > keeping up a decent release cadence but we have an abysmal track record
> > when it comes to pushing out majors. First we were afraid to commit --
> > witness how long it took us to get to a 1.0 -- and then pushing out the
> 1.0
> > took a monster effort. 2.0 looks to be a repeat of the errors of 1.0. My
> > sense is that 2.0 is beyond saving at this stage.
> >
> > Can we do 3.0 different? As per your suggestion?
> >
> > St.Ack
> >
>
>
>
> --
> Best regards,
>
>- Andy
>
> If you are given a choice, you believe you have acted freely. - Raymond
> Teller (via Peter Watts)
>



-- 
Thanks and Regards,
Ashu Pachauri

[jira] [Created] (HBASE-17766) Generate Javadoc for hbase-spark module

[Yi Liang (JIRA)]

Yi Liang created HBASE-17766:


 Summary: Generate Javadoc for hbase-spark module 
 Key: HBASE-17766
 URL: https://issues.apache.org/jira/browse/HBASE-17766
 Project: HBase
  Issue Type: Bug
Reporter: Yi Liang
Assignee: Yi Liang
 Fix For: 2.0.0


 Scala classes in hbase-spark module aren't showing up in our API docs nor our 
internal API docs. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: A suggestion for releasing major versions faster (Was: NOTICE: Nice testimony on benefits of the offheap read-path work now up on our blog)

[Andrew Purtell]

The only way forward for saving 2.0 at this point is to *make the branch
and spin the RC. *Just do it. Kick out by revert what obviously isn't
ready. Solicit help in getting partially finished things into working
state. Kick them out too if the help does not arrive.

Maybe too much is in a half done state and the scale of effort for those
reverts is too high. Fine. Renumber master as 3.0, and make a branch-2 from
branch-1 and backport a select number of things from master into the new
branch-2. Then release. I do a variation of this for the $dayjob so would
be your man to turn to for driving this if that's the way forward.

I know it's easy to recommend the labor of others. Depending on what we are
going to do I can talk to work about freeing up time to help.


On Thu, Mar 9, 2017 at 11:18 AM, Stack  wrote:

> On Thu, Mar 9, 2017 at 12:34 AM, Phil Yang  wrote:
> >
> >
> > So my suggestion is cutting branch-x faster and have some fixed period,
> for
> > example, six month or one year?
> >
>
>
> You are right Phil.
>
> The Release Managers for the minor releases have been doing a good job
> keeping up a decent release cadence but we have an abysmal track record
> when it comes to pushing out majors. First we were afraid to commit --
> witness how long it took us to get to a 1.0 -- and then pushing out the 1.0
> took a monster effort. 2.0 looks to be a repeat of the errors of 1.0. My
> sense is that 2.0 is beyond saving at this stage.
>
> Can we do 3.0 different? As per your suggestion?
>
> St.Ack
>



-- 
Best regards,

   - Andy

If you are given a choice, you believe you have acted freely. - Raymond
Teller (via Peter Watts)

Re: A suggestion for releasing major versions faster (Was: NOTICE: Nice testimony on benefits of the offheap read-path work now up on our blog)

[Stack]

On Thu, Mar 9, 2017 at 12:34 AM, Phil Yang  wrote:
>
>
> So my suggestion is cutting branch-x faster and have some fixed period, for
> example, six month or one year?
>


You are right Phil.

The Release Managers for the minor releases have been doing a good job
keeping up a decent release cadence but we have an abysmal track record
when it comes to pushing out majors. First we were afraid to commit --
witness how long it took us to get to a 1.0 -- and then pushing out the 1.0
took a monster effort. 2.0 looks to be a repeat of the errors of 1.0. My
sense is that 2.0 is beyond saving at this stage.

Can we do 3.0 different? As per your suggestion?

St.Ack

[jira] [Created] (HBASE-17765) Reviving the merge possibility in the CompactingMemStore

[Anastasia Braginsky (JIRA)]

Anastasia Braginsky created HBASE-17765:
---

 Summary: Reviving the merge possibility in the CompactingMemStore
 Key: HBASE-17765
 URL: https://issues.apache.org/jira/browse/HBASE-17765
 Project: HBase
  Issue Type: Sub-task
Reporter: Anastasia Braginsky


According to the new performance results presented in the HBASE-16417 we see 
that the read latency of the 90th percentile of the BASIC policy is too big due 
to the need to traverse through too many segments in the pipeline. In this JIRA 
we correct the bug in the merge sizing calculations and allow pipeline size 
threshold to be a configurable parameter.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HBASE-17764) Solve TestMultiSlaveReplication flakiness

[Stephen Yuan Jiang (JIRA)]

Stephen Yuan Jiang created HBASE-17764:
--

 Summary: Solve TestMultiSlaveReplication flakiness 
 Key: HBASE-17764
 URL: https://issues.apache.org/jira/browse/HBASE-17764
 Project: HBase
  Issue Type: Bug
  Components: test
Reporter: Stephen Yuan Jiang
Assignee: Stephen Yuan Jiang
Priority: Minor


When I doing testing for a feature I am working on, the 
TestMultiSlaveReplication test was flaky.  
TestMultiSlaveReplication#testMultiSlaveReplication would always pass when 
running alone; however, it would fail occasionally when running with the suite 
(the other test TestMultiSlaveReplication#testZKLockCleaner always run first 
and I suspect that it withhold some zookeeper resource, which caused the 
problem).  

After I set start and shutdown zk cluster before and after each test, I no 
longer see the problem.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: What if I want to write another table in MasterObserver.preModifyColumnFamily ?

[Ted Yu]

Zookeeper is supposed to store transient data.
Plus you need to secure the audit log in zookeeper so that unauthorized
user cannot temper the audit.

On Thu, Mar 9, 2017 at 9:24 AM, Xi Yang  wrote:

> Thank you Anoop. After thinking, I give up the idea about write records to
> RS, I will using other ways like zk, or just a text file. I appreciate your
> help!
>
> 2017-03-07 23:36 GMT-08:00 Anoop John :
>
> > It will be cross server call.. The pre/postModifyColumn is in master
> > side. (MasterObserver)  You want to write the info to a table means
> > this will be in some RS.  So an RPC request will be needed. But still
> > it is not a case like one RS to another RS cross server call where
> > there is a remote chance of all handlers getting stuck and possible
> > deadlock.  Need to carefully done though!
> >
> > -Anoop-
> >
> > On Wed, Mar 8, 2017 at 8:26 AM, Xi Yang  wrote:
> > > Got it. I appreciate your help!
> > >
> > > 2017-03-07 17:05 GMT-08:00 Ted Yu :
> > >
> > >> It seems the following hook is better for your use case:
> > >>
> > >>   default void postModifyColumn(final
> > >> ObserverContext ctx,
> > >>   TableName tableName, HColumnDescriptor columnFamily) throws
> > >> IOException {}
> > >>
> > >> since there is no guarantee that column family is modified at
> > >> time preModifyColumnFamily() is called.
> > >>
> > >> Cheers
> > >>
> > >> On Tue, Mar 7, 2017 at 4:43 PM, Xi Yang 
> wrote:
> > >>
> > >> > Requirement:
> > >> >
> > >> > I want to record every change of modify columnFamily by using
> > >> > preModifyColumnFamily().
> > >> > Now I have a table "my_ddl_log" which used to record the change of
> > >> > columnFamily. For example:
> > >> >
> > >> > If jack change the TTL of columnFamily "primary" in table
> "employee".
> > >> Then
> > >> > we should add a put to "my_ddl_log" like this record:
> > >> > log:name= 'jack'
> > >> > log:updateTime= '2017-03-07 12:12 GMT-08:00'
> > >> > log:change= ''Change TTL of Table: employee ColumnFamily: primary'
> > >> >
> > >> > I try to use preModifyColumnFamily to do this stuff.
> > >> >
> > >> > Thanks,
> > >> > Alex
> > >> >
> > >> >
> > >> > 2017-03-07 12:12 GMT-08:00 Ted Yu :
> > >> >
> > >> > > Describing your use case would allow people to give better answer.
> > >> > >
> > >> > > What kind of data do you write to other table in
> > >> preModifyColumnFamily()
> > >> > ?
> > >> > >
> > >> > > Cross server call within observer is not good idea.
> > >> > >
> > >> > > Take a look at ConnectionUtils.createShortCircuitConnection().
> > >> > >
> > >> > > Cheers
> > >> > >
> > >> > > On Tue, Mar 7, 2017 at 11:42 AM, Xi Yang 
> > >> wrote:
> > >> > >
> > >> > > > All the articles I've ever seen are talking about add
> > increment
> > >> or
> > >> > > > change put/get status or pinrt out logs. what if I want to write
> > some
> > >> > > data
> > >> > > > to another table in Observer? For
> > >> > > > example, MasterObserver.preModifyColumnFamily()? Seems Observer
> > is
> > >> > > runing
> > >> > > > at server side, so use connection is unneccessary and might
> raise
> > >> some
> > >> > > > problem.
> > >> > > >I know this might be a stupid question, so if you can just
> give
> > >> some
> > >> > > > links let me to learn without explain in email, I will be
> > grateful to
> > >> > you
> > >> > > > for your help
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Alex
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>

Re: [VOTE] Backup/Restore feature for HBase 2.0, vote closing 3/11/2017

[Vladimir Rodionov]

bump

On Wed, Mar 8, 2017 at 3:52 PM, Vladimir Rodionov 
wrote:

> No problem, we can extend deadline.
>
> On Wed, Mar 8, 2017 at 3:31 PM, Ted Yu  wrote:
>
>> March 11th is on weekend.
>>
>> Do you want to give people who haven't looked at the mega patch in depth
>> some more time ?
>>
>> Cheers
>>
>> On Wed, Mar 8, 2017 at 1:32 PM, Vladimir Rodionov > >
>> wrote:
>>
>> > Hello, HBase folks
>> >
>> > For your consideration today is Backup/Restore feature for Apache HBAse
>> > 2.0.
>> > Backup code is available as a mega patch in HBASE-14123 (v61), applies
>> > cleanly to the current master, all test PASS, patch has no other issues.
>> >
>> > The patch has gone through numerous rounds of code reviews and has
>> probably
>> > the most lengthy discussion thread on Apache JIRA (HBASE-14123) :)
>> >
>> > The work has been split into 3 phases (HBASE-14030, 14123, 14414) Two
>> first
>> > are complete, third one is still in progress.
>> >
>> >
>> > *** Summary of work HBASE-14123
>> >
>> > The new feature introduces new command-line extensions to the hbase
>> command
>> > and, from the client side, is accessible through command-line only
>> > Operations:
>> > * Create full backup on a list of tables or backup set
>> > * Create incremental backup image for table list or backup set
>> > * Restore list of tables from a given backup image
>> > * Show current backup progress
>> > * Delete backup image and all related images
>> > * Show history of backups
>> > * Backup set operations: create backup set, add/remove table to/from
>> backup
>> > set, etc
>> >
>> > In the current implementation, the feature is already usable, meaning
>> that
>> > users can backup tables and restore them using provided command-line
>> tools.
>> > Both: full and incremental backups are supported.
>> > This work is based on original work of IBM team (HBASE-7912). The full
>> list
>> > of JIRAs included in this mega patch can be found in three umbrella
>> JIRAs:
>> > HBASE-14030 (Phase 1), HBASE-14123 (Phase 2) and HBASE-14414 (Phase 3 -
>> all
>> > resolved ones made it into the patch)
>> >
>> > *** What are the remaining work items
>> >
>> > All remaining items can be found in Phase 3 umbrella JIRA: HBASE-14414.
>> > They are split into 3 groups: BLOCKER, CRITICAL, MAJOR
>> > Only BLOCKERs and CRITICALs are guaranteed for HBase 2.0 release.
>> >
>> > * BLOCKER
>> >
>> > * HBASE-14417 Incremental backup and bulk loading ( Patch available)
>> > * HBASE-14135 HBase Backup/Restore Phase 3: Merge backup images
>> > * HBASE-14141 HBase Backup/Restore Phase 3: Filter WALs on backup to
>> > include only edits from backup tables (Patch available)
>> > * HBASE-17133 Backup documentation
>> > * HBASE-15227 Fault tolerance support
>> >
>> > * CRITICAL
>> >
>> > * HBASE-16465 Disable split/merges during backup
>> >
>> > We have umbrella JIRA (HBASE-14414) to track all the remaining work
>> > All the BLOCKER and CRITICAL JIRAs currently in open state will be
>> > implemented by 2.0 release time. Some MAJOR too, but it depends on
>> resource
>> > availability
>> > The former development branch (HBASE-7912) is obsolete and will be
>> > closed/deleted after the merge.
>> > We want backup to be a GA feature in 2.0
>> > We are going to support full backward compatibility for backup tool in
>> 2.0
>> > and onwards.
>> >
>> >  Configuration
>> >
>> > Backup is disabled, by default. To enable it, the following
>> configuration
>> > properties must be added to hbase-site.xml:
>> >
>> > hbase.backup.enable=true
>> > hbase.master.logcleaner.plugins=YOUR_PLUGINS,org.
>> > apache.hadoop.hbase.backup.master.BackupLogCleaner
>> > hbase.procedure.master.classes=YOUR_CLASSES,org.
>> > apache.hadoop.hbase.backup.master.LogRollMasterProcedureManager
>> > hbase.procedure.regionserver.classes=YOUR_CLASSES,org.
>> > apache.hadoop.hbase.backup.regionserver.LogRollRegionServerProcedureMa
>> > nager
>> >
>> >
>> > I would like to thank IBM team and Jerry He for original work,
>> >
>> > Enis, Ted, Stack, Matteo, Jerry for time spent on code reviews
>> >
>> > Special thanks to Ted Yu for his co-development work.
>> >
>> > References:
>> >
>> > https://issues.apache.org/jira/browse/HBASE-7912 (original IBM,
>> contains
>> > design doc)
>> > https://issues.apache.org/jira/browse/HBASE-14030 (Phase 1)
>> > https://issues.apache.org/jira/browse/HBASE-14123 (Phase 2)
>> > https://issues.apache.org/jira/browse/HBASE-14414 (Phase 3)
>> >
>> > Please  vote +1/-1 by midnight Pacific Time (00:00
>> > -0800 GMT) on March 11th  ​on whether or not we should merge this into
>> the
>> > current master.
>> >
>> > -Vladimir Rodionov
>> >
>>
>
>

Re: [VOTE] Backup/Restore feature for HBase 2.0, vote closing 3/11/2017

[Ted Yu]

+1 from me as well.

On Wed, Mar 8, 2017 at 3:48 PM, Enis Söztutar  wrote:

> Thanks Vladimir for the write up and the work. Glad to see progress.
>
> Here is my +1. I'm pretty sure we can get the blockers in before the 2.0
> timeframe with the momentum, so it is a good idea to merge now so that
> development can continue in master, and there is more exposure for testing,
> etc.
>
> Enis
>
> On Wed, Mar 8, 2017 at 1:32 PM, Vladimir Rodionov 
> wrote:
>
> > Hello, HBase folks
> >
> > For your consideration today is Backup/Restore feature for Apache HBAse
> > 2.0.
> > Backup code is available as a mega patch in HBASE-14123 (v61), applies
> > cleanly to the current master, all test PASS, patch has no other issues.
> >
> > The patch has gone through numerous rounds of code reviews and has
> probably
> > the most lengthy discussion thread on Apache JIRA (HBASE-14123) :)
> >
> > The work has been split into 3 phases (HBASE-14030, 14123, 14414) Two
> first
> > are complete, third one is still in progress.
> >
> >
> > *** Summary of work HBASE-14123
> >
> > The new feature introduces new command-line extensions to the hbase
> command
> > and, from the client side, is accessible through command-line only
> > Operations:
> > * Create full backup on a list of tables or backup set
> > * Create incremental backup image for table list or backup set
> > * Restore list of tables from a given backup image
> > * Show current backup progress
> > * Delete backup image and all related images
> > * Show history of backups
> > * Backup set operations: create backup set, add/remove table to/from
> backup
> > set, etc
> >
> > In the current implementation, the feature is already usable, meaning
> that
> > users can backup tables and restore them using provided command-line
> tools.
> > Both: full and incremental backups are supported.
> > This work is based on original work of IBM team (HBASE-7912). The full
> list
> > of JIRAs included in this mega patch can be found in three umbrella
> JIRAs:
> > HBASE-14030 (Phase 1), HBASE-14123 (Phase 2) and HBASE-14414 (Phase 3 -
> all
> > resolved ones made it into the patch)
> >
> > *** What are the remaining work items
> >
> > All remaining items can be found in Phase 3 umbrella JIRA: HBASE-14414.
> > They are split into 3 groups: BLOCKER, CRITICAL, MAJOR
> > Only BLOCKERs and CRITICALs are guaranteed for HBase 2.0 release.
> >
> > * BLOCKER
> >
> > * HBASE-14417 Incremental backup and bulk loading ( Patch available)
> > * HBASE-14135 HBase Backup/Restore Phase 3: Merge backup images
> > * HBASE-14141 HBase Backup/Restore Phase 3: Filter WALs on backup to
> > include only edits from backup tables (Patch available)
> > * HBASE-17133 Backup documentation
> > * HBASE-15227 Fault tolerance support
> >
> > * CRITICAL
> >
> > * HBASE-16465 Disable split/merges during backup
> >
> > We have umbrella JIRA (HBASE-14414) to track all the remaining work
> > All the BLOCKER and CRITICAL JIRAs currently in open state will be
> > implemented by 2.0 release time. Some MAJOR too, but it depends on
> resource
> > availability
> > The former development branch (HBASE-7912) is obsolete and will be
> > closed/deleted after the merge.
> > We want backup to be a GA feature in 2.0
> > We are going to support full backward compatibility for backup tool in
> 2.0
> > and onwards.
> >
> >  Configuration
> >
> > Backup is disabled, by default. To enable it, the following configuration
> > properties must be added to hbase-site.xml:
> >
> > hbase.backup.enable=true
> > hbase.master.logcleaner.plugins=YOUR_PLUGINS,org.
> > apache.hadoop.hbase.backup.master.BackupLogCleaner
> > hbase.procedure.master.classes=YOUR_CLASSES,org.
> > apache.hadoop.hbase.backup.master.LogRollMasterProcedureManager
> > hbase.procedure.regionserver.classes=YOUR_CLASSES,org.
> > apache.hadoop.hbase.backup.regionserver.LogRollRegionServerProcedureMa
> > nager
> >
> >
> > I would like to thank IBM team and Jerry He for original work,
> >
> > Enis, Ted, Stack, Matteo, Jerry for time spent on code reviews
> >
> > Special thanks to Ted Yu for his co-development work.
> >
> > References:
> >
> > https://issues.apache.org/jira/browse/HBASE-7912 (original IBM, contains
> > design doc)
> > https://issues.apache.org/jira/browse/HBASE-14030 (Phase 1)
> > https://issues.apache.org/jira/browse/HBASE-14123 (Phase 2)
> > https://issues.apache.org/jira/browse/HBASE-14414 (Phase 3)
> >
> > Please  vote +1/-1 by midnight Pacific Time (00:00
> > -0800 GMT) on March 11th  ​on whether or not we should merge this into
> the
> > current master.
> >
> > -Vladimir Rodionov
> >
>

[jira] [Reopened] (HBASE-17761) Test TestRemoveRegionMetrics.testMoveRegion fails intermittently because of race condition

[Umesh Agashe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Umesh Agashe reopened HBASE-17761:
--

Adding branch-1 patch

> Test TestRemoveRegionMetrics.testMoveRegion fails intermittently because of 
> race condition
> --
>
> Key: HBASE-17761
> URL: https://issues.apache.org/jira/browse/HBASE-17761
> Project: HBase
>  Issue Type: Bug
>  Components: Region Assignment
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
> Fix For: 2.0.0
>
> Attachments: HBASE-17761.patch, HBASE-17761.v1.patch
>
>
> After moving the region code waits till all regions are assigned but not for 
> region to go online on a destination server. On branch-1 TestAdmin1.java has 
> a function moveRegionAndWait() that moves the region and waits for the region 
> to become online on destination server. We need to use that function in this 
> test.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: What if I want to write another table in MasterObserver.preModifyColumnFamily ?

[Xi Yang]

Thank you Anoop. After thinking, I give up the idea about write records to
RS, I will using other ways like zk, or just a text file. I appreciate your
help!

2017-03-07 23:36 GMT-08:00 Anoop John :

> It will be cross server call.. The pre/postModifyColumn is in master
> side. (MasterObserver)  You want to write the info to a table means
> this will be in some RS.  So an RPC request will be needed. But still
> it is not a case like one RS to another RS cross server call where
> there is a remote chance of all handlers getting stuck and possible
> deadlock.  Need to carefully done though!
>
> -Anoop-
>
> On Wed, Mar 8, 2017 at 8:26 AM, Xi Yang  wrote:
> > Got it. I appreciate your help!
> >
> > 2017-03-07 17:05 GMT-08:00 Ted Yu :
> >
> >> It seems the following hook is better for your use case:
> >>
> >>   default void postModifyColumn(final
> >> ObserverContext ctx,
> >>   TableName tableName, HColumnDescriptor columnFamily) throws
> >> IOException {}
> >>
> >> since there is no guarantee that column family is modified at
> >> time preModifyColumnFamily() is called.
> >>
> >> Cheers
> >>
> >> On Tue, Mar 7, 2017 at 4:43 PM, Xi Yang  wrote:
> >>
> >> > Requirement:
> >> >
> >> > I want to record every change of modify columnFamily by using
> >> > preModifyColumnFamily().
> >> > Now I have a table "my_ddl_log" which used to record the change of
> >> > columnFamily. For example:
> >> >
> >> > If jack change the TTL of columnFamily "primary" in table "employee".
> >> Then
> >> > we should add a put to "my_ddl_log" like this record:
> >> > log:name= 'jack'
> >> > log:updateTime= '2017-03-07 12:12 GMT-08:00'
> >> > log:change= ''Change TTL of Table: employee ColumnFamily: primary'
> >> >
> >> > I try to use preModifyColumnFamily to do this stuff.
> >> >
> >> > Thanks,
> >> > Alex
> >> >
> >> >
> >> > 2017-03-07 12:12 GMT-08:00 Ted Yu :
> >> >
> >> > > Describing your use case would allow people to give better answer.
> >> > >
> >> > > What kind of data do you write to other table in
> >> preModifyColumnFamily()
> >> > ?
> >> > >
> >> > > Cross server call within observer is not good idea.
> >> > >
> >> > > Take a look at ConnectionUtils.createShortCircuitConnection().
> >> > >
> >> > > Cheers
> >> > >
> >> > > On Tue, Mar 7, 2017 at 11:42 AM, Xi Yang 
> >> wrote:
> >> > >
> >> > > > All the articles I've ever seen are talking about add
> increment
> >> or
> >> > > > change put/get status or pinrt out logs. what if I want to write
> some
> >> > > data
> >> > > > to another table in Observer? For
> >> > > > example, MasterObserver.preModifyColumnFamily()? Seems Observer
> is
> >> > > runing
> >> > > > at server side, so use connection is unneccessary and might raise
> >> some
> >> > > > problem.
> >> > > >I know this might be a stupid question, so if you can just give
> >> some
> >> > > > links let me to learn without explain in email, I will be
> grateful to
> >> > you
> >> > > > for your help
> >> > > >
> >> > > > Thanks,
> >> > > > Alex
> >> > > >
> >> > >
> >> >
> >>
>

Successful: HBase Generate Website

[Apache Jenkins Server]

Build status: Successful

If successful, the website and docs have been generated. To update the live 
site, follow the instructions below. If failed, skip to the bottom of this 
email.

Use the following commands to download the patch and apply it to a clean branch 
based on origin/asf-site. If you prefer to keep the hbase-site repo around 
permanently, you can skip the clone step.

  git clone https://git-wip-us.apache.org/repos/asf/hbase-site.git

  cd hbase-site
  wget -O- 
https://builds.apache.org/job/hbase_generate_website/511/artifact/website.patch.zip
 | funzip > 1160315e2f666fd5c51505f55f77399e302885f2.patch
  git fetch
  git checkout -b asf-site-1160315e2f666fd5c51505f55f77399e302885f2 
origin/asf-site
  git am --whitespace=fix 1160315e2f666fd5c51505f55f77399e302885f2.patch

At this point, you can preview the changes by opening index.html or any of the 
other HTML pages in your local 
asf-site-1160315e2f666fd5c51505f55f77399e302885f2 branch.

There are lots of spurious changes, such as timestamps and CSS styles in 
tables, so a generic git diff is not very useful. To see a list of files that 
have been added, deleted, renamed, changed type, or are otherwise interesting, 
use the following command:

  git diff --name-status --diff-filter=ADCRTXUB origin/asf-site

To see only files that had 100 or more lines changed:

  git diff --stat origin/asf-site | grep -E '[1-9][0-9]{2,}'

When you are satisfied, publish your changes to origin/asf-site using these 
commands:

  git commit --allow-empty -m "Empty commit" # to work around a current ASF 
INFRA bug
  git push origin asf-site-1160315e2f666fd5c51505f55f77399e302885f2:asf-site
  git checkout asf-site
  git branch -D asf-site-1160315e2f666fd5c51505f55f77399e302885f2

Changes take a couple of minutes to be propagated. You can verify whether they 
have been propagated by looking at the Last Published date at the bottom of 
http://hbase.apache.org/. It should match the date in the index.html on the 
asf-site branch in Git.

As a courtesy- reply-all to this email to let other committers know you pushed 
the site.



If failed, see https://builds.apache.org/job/hbase_generate_website/511/console

Re: HDFS Balancer

[Jean-Marc Spaggiari]

So there is no way to use the pinning feature without having to use the
favored nodes option? :(

Le 2017-03-08 6:13 AM, "Harsh J"  a écrit :

> Hey Lars!,
>
> I was on a similar line of investigation today, and I've filed
> https://issues.apache.org/jira/browse/HBASE-17760 to change the text. The
> pinning part of the text is relevant, but the command part isn't. In
> addition, you'd need to manually use the FavoredNodeLoadBalancer work to
> actually get HBase to apply pinning to its writes by passing around proper
> favored-node hint hostnames. I've also linked past and future relevant work
> JIRAs to that one.
>
> Stumbled on this email when searching some follow-throughs, thought I'd
> drop a note.
>
> On Tue, 7 Mar 2017 at 20:18 Ted Yu  wrote:
>
> > bq. how that - apparently wrong - information came about
> >
> > Maybe Sean / Misty can give some context.
> >
> > Cheers
> >
> > On Tue, Mar 7, 2017 at 6:37 AM, Lars George 
> wrote:
> >
> > > Hey Ted,
> > >
> > > Thanks Cpt. Obvious :)
> > >
> > > I know how to use "blame" or git log how to find the JIRA, but what I
> was
> > > after is how that - apparently wrong - information came about. And if
> it
> > is
> > > wrong, what _is_ the current status of this feature.
> > >
> > > I do believe this is an important operational piece as it helps with
> > > rearranging clusters. Since it seems to still be missing, I am
> wondering
> > > what needs to be done here.
> > >
> > > Makes sense?
> > >
> > > Lars
> > >
> > > Sent from my iPhone
> > >
> > > > On 6 Mar 2017, at 19:50, Ted Yu  wrote:
> > > >
> > > > w.r.t. the first question, the quoted paragraph came from:
> > > >
> > > > HBASE-15332 Document how to take advantage of HDFS-6133 in HBase
> > > >
> > > >> On Mon, Mar 6, 2017 at 6:38 PM, Lars George 
> > > wrote:
> > > >>
> > > >> Hi,
> > > >>
> > > >> I am trying to grok what came out of all these issues about the HDFS
> > > >> balancer and being able to avoid it destroying HBase locality. There
> > > >> is this https://issues.apache.org/jira/browse/HBASE-13021 from JM,
> > and
> > > >> the book http://hbase.apache.org/book.html#_hbase_and_hdfs refers
> to
> > > >> https://issues.apache.org/jira/browse/HDFS-6133, stating:
> > > >>
> > > >> "HDFS-6133 provides the ability to exclude a given directory from
> the
> > > >> HDFS load balancer, by setting the dfs.datanode.block-pinning.
> enabled
> > > >> property to true in your HDFS configuration and running the
> following
> > > >> hdfs command:
> > > >>
> > > >> $ sudo -u hdfs hdfs balancer -exclude /hbase"
> > > >>
> > > >> I checked the Balancer class in 2.7.2 and it does not have that
> > > >> support, i.e. being able to exclude a path, it can only exclude
> hosts.
> > > >> That is also clear from HDFS-6133, which adds favoured nodes, but
> not
> > > >> being able to exclude paths (which would be nice).
> > > >>
> > > >> HBASE-13021 mentions that this works in tandem with the HBase
> favored
> > > >> node feature, but that makes it much more complicated since you have
> > > >> to pin individual regions to nodes, instead of doing that wholesale.
> > > >>
> > > >> Where does the above in the HBase book come from, and what is the
> > > >> current state as far as you know?
> > > >>
> > > >> Cheers,
> > > >> Lars
> > > >>
> > >
> >
>

Re: A suggestion for releasing major versions faster (Was: NOTICE: Nice testimony on benefits of the offheap read-path work now up on our blog)

[Duo Zhang]

Yeah we have put too much things into the 2.0 release. And I believe there
is a release schedule for 2.0 in the mailing-list?

2017-03-09 16:34 GMT+08:00 Phil Yang :

> Nice article! Thanks for the authors.
>
> An off-the-topic suggestion to dev list:
>
> Now we commit some big and new features to master branch only, which will
> be included in a new major version, for example, 2.0. However, a major
> version is not upgraded very quickly, 1.0.0 was released two years ago and
> we still don't release 2.0.0. HBASE-11425 is resolved more than one year
> ago. In the worst case, if we make some work done and commit to master
> branch just after cutting a new major release branch, the feature will be
> released two years later or even longer. It is no need and no benefits to
> wait so long. Bugs of the new feature in dev branches only are hardly to be
> found, and committers may not be familiar with a feature which is committed
> two years ago...
>
> So my suggestion is cutting branch-x faster and have some fixed period, for
> example, six month or one year?
>
> Thanks,
> Phil
>
>
> 2017-03-09 14:24 GMT+08:00 Stack :
>
> > See writeup on how work done by your Yu Li, Sun Yu, Anoop Sam John, and
> > Ramkrishna S Vasudevan improved throughput at scale on Singles Day 2016 @
> > Alibaba.
> >
> > Read all about it at https://blogs.apache.org/hbase/
> >
> > Yours,
> > St.Ack
> >
>

[jira] [Created] (HBASE-17763) IPCUtil.wrapException will wrap DoNotRetryIOException with IOException

[Duo Zhang (JIRA)]

Duo Zhang created HBASE-17763:
-

 Summary: IPCUtil.wrapException will wrap DoNotRetryIOException 
with IOException
 Key: HBASE-17763
 URL: https://issues.apache.org/jira/browse/HBASE-17763
 Project: HBase
  Issue Type: Bug
  Components: IPC/RPC
Affects Versions: 2.0.0
Reporter: Duo Zhang
 Fix For: 2.0.0






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

A suggestion for releasing major versions faster (Was: NOTICE: Nice testimony on benefits of the offheap read-path work now up on our blog)

[Phil Yang]

Nice article! Thanks for the authors.

An off-the-topic suggestion to dev list:

Now we commit some big and new features to master branch only, which will
be included in a new major version, for example, 2.0. However, a major
version is not upgraded very quickly, 1.0.0 was released two years ago and
we still don't release 2.0.0. HBASE-11425 is resolved more than one year
ago. In the worst case, if we make some work done and commit to master
branch just after cutting a new major release branch, the feature will be
released two years later or even longer. It is no need and no benefits to
wait so long. Bugs of the new feature in dev branches only are hardly to be
found, and committers may not be familiar with a feature which is committed
two years ago...

So my suggestion is cutting branch-x faster and have some fixed period, for
example, six month or one year?

Thanks,
Phil


2017-03-09 14:24 GMT+08:00 Stack :

> See writeup on how work done by your Yu Li, Sun Yu, Anoop Sam John, and
> Ramkrishna S Vasudevan improved throughput at scale on Singles Day 2016 @
> Alibaba.
>
> Read all about it at https://blogs.apache.org/hbase/
>
> Yours,
> St.Ack
>