[jira] [Created] (HBASE-15515) Improve LocalityBasedCandidateGenerator in Balancer
Guanghao Zhang created HBASE-15515: -- Summary: Improve LocalityBasedCandidateGenerator in Balancer Key: HBASE-15515 URL: https://issues.apache.org/jira/browse/HBASE-15515 Project: HBase Issue Type: Bug Affects Versions: 2.0.0, 1.3.0 Reporter: Guanghao Zhang Assignee: Guanghao Zhang Priority: Minor Fix For: 2.0.0 There are some problems which need to fix. 1. LocalityBasedCandidateGenerator.getLowestLocalityRegionOnServer should skip empty region. 2. When use LocalityBasedCandidateGenerator to generate Cluster.Action, it should add random operation instead of pickLowestLocalityServer(cluster). Because the search function may stuck here if it always generate the same Cluster.Action. 3. getLeastLoadedTopServerForRegion should get least loaded server which have better locality than current server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15514) Single (or small number) of Column Family in Mutation optimization
Vladimir Rodionov created HBASE-15514: - Summary: Single (or small number) of Column Family in Mutation optimization Key: HBASE-15514 URL: https://issues.apache.org/jira/browse/HBASE-15514 Project: HBase Issue Type: Improvement Components: Performance Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov familyMap in Mutation class is TreeMap>. Overhead of TreeMap - 48 bytes, TreeMap$Entry - 40 bytes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Branch for 1.3
At least include HBASE-15400 in 1.3 please... Otherwise these things can only be integrated in 1.4 since DTCP can not be used together with DefaultStoreEngine after HBASE-15400 getting in. 2016-03-22 2:03 GMT+08:00 Enis Söztutar : > You may want to track https://issues.apache.org/jira/browse/HBASE-15339 as > a parent for date-tiered compaction improvements. Current compaction policy > is useful in its own, but handling existing data, bulk loading etc will be > improved with these subtasks. I think the patches can land before the 1.3 > timeframe, but of course open for discussion for inclusion. My vote would > be to include all the improvements since it would be easier to tell the > story to users. > > Enis > > On Mon, Mar 21, 2016 at 3:32 AM, Ted Yu wrote: > > > For Spark connector, we should start a separate discussion thread about > > backporting to branch-1. > > > > Zhan has a bug fix coming this week which deals with how negative numbers > > are handled in comparison. > > > > FYI > > > > On Mon, Mar 21, 2016 at 1:35 AM, Mikhail Antonov > > wrote: > > > > > Hi folks, > > > > > > bringing this topic up again. I'm planning to start spinning 1.3 builds > > and > > > see if/where they break in a week or two, and (depending on how it > does) > > > start preparing RCs in a month or maybe two. So, let's see where we > are. > > > > > > Big items first. There were long debates around three big items - MOBs, > > > Spark connector and RS groups, whether we should have them or not. > > > > > > - MOBs > > > I believe we decided that they aren't going to go in branch-1, and > hence > > > not in branch-1.3 for sure. We might get back to that debate and > > > re-consider MOBs for branch-1 if 2.0 is delayed, but almost certainly > > they > > > won't make it in 1.3 timeframe anyway as I feel. > > > > > > - Spark connector - HBASE-14160 it looks like it has 3 subtasks open, > one > > > of which is big one (HBASE-14375) - define public API for Spark > > > integration. From the Jira looks like active work is happening on other > > > subtasks, but not on this one. So how's public API going? How stable it > > is? > > > Who would want to have Spark in 1.3 and willing to help with this? > OTOH - > > > who has objections about back-porting it? Has anyone been using it in > > some > > > real environment? > > > > > > - RS groups - there was recently a thread about them, I'd like to > bring > > it > > > up again and get to some conclusion. > > > > > > Other features which we had in flight a month ago - > > > > > > - HBASE-15181 - date based tiered compactions has landed > > > - HBASE-11290 - unlock RegionStates. I'm afraid the codebase has moved > > > forward quite a bit since the benchmark was run on this change :( - > > Francis > > > - are you using it now? If we could have some benchmarks on the latest > > > rebase that I think would be great. > > > > > > > > > - HBASE-13557 - special handling for system tables WALs - should we > > still > > > keep it targeted for 1.3? > > > - HBASE-13017 - keep table state in meta, this one doesn't look like > > it's > > > going to make it in > > > > > > As a new item on my list - I'm looking forward to see more of > HBASE-15492 > > > (memory optimizations) subtasks to go in branch-1. > > > > > > Thanks! > > > Mikhail > > > > > > On Sun, Feb 28, 2016 at 10:53 AM, Andrew Purtell < > > andrew.purt...@gmail.com > > > > > > > wrote: > > > > > > > I'm starting a push at work to get us up on 1.2. Assuming that > happens > > > > later this year I think that will be the end of my close attention to > > > 0.98. > > > > > > > > > On Feb 26, 2016, at 1:54 PM, Nick Dimiduk > > wrote: > > > > > > > > > > On Fri, Feb 26, 2016 at 12:23 PM, Andrew Purtell < > > apurt...@apache.org> > > > > > wrote: > > > > > > > > > >> In the meantime those of us running HBase in production would > > benefit > > > > from > > > > >> fairly frequent minor releases. > > > > > > > > > > +1. Having to look back to 0.98 to get some new feature is > > problematic. > > > > > > > > > >> On Fri, Feb 26, 2016 at 9:50 AM, Elliott Clark > > > > > wrote: > > > > >> > > > > >>> I disagree. We have agreed that 2.0 will have a new assignement > > > > manager. > > > > >>> There's a lot of work that has been done on getting that in, so > far > > > > there > > > > >>> are no benefits to the end user from all that work. We should > stick > > > > with > > > > >>> the plan and release 2.0 when it's ready. > > > > >>> > > > > >>> On Fri, Feb 26, 2016 at 9:39 AM, Stephen Jiang < > > > > syuanjiang...@gmail.com> > > > > >>> wrote: > > > > >>> > > > > Thanks for Mikhail for taking the 1.3 RM role. Looks like we > > have a > > > > >> lot > > > > >>> of > > > > new things in 1.3 release. > > > > > > > > Based on the experience of 1.1 and 1.2 release, it takes a lot > of > > > > >> efforts > > > > to get a stable minor release out. From this, I have my own > > 2-cents > > > > on > > > > >>> 1.4 > > > > release.
[jira] [Reopened] (HBASE-15360) Fix flaky TestSimpleRpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-15360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reopened HBASE-15360: > Fix flaky TestSimpleRpcScheduler > > > Key: HBASE-15360 > URL: https://issues.apache.org/jira/browse/HBASE-15360 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 2.0.0, 1.3.0 >Reporter: Mikhail Antonov >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-15360.patch > > > There were several flaky tests added there as part of HBASE-15306 and likely > HBASE-15136. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15513) hbase.hregion.memstore.chunkpool.maxsize is 0.0 by default
Vladimir Rodionov created HBASE-15513: - Summary: hbase.hregion.memstore.chunkpool.maxsize is 0.0 by default Key: HBASE-15513 URL: https://issues.apache.org/jira/browse/HBASE-15513 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 2.0.0 That results in excessive MemStoreLAB chunk allocations because we can not reuse them. Not sure, why it has been disabled, by default. May be the code has not been tested well? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15512) Avoid cell allocation on Store.add(Cell)
Vladimir Rodionov created HBASE-15512: - Summary: Avoid cell allocation on Store.add(Cell) Key: HBASE-15512 URL: https://issues.apache.org/jira/browse/HBASE-15512 Project: HBase Issue Type: Improvement Components: Performance Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov AbstractMemStore.add(Cell) clones the cell. No reason for that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15511) ClusterStatus should be able
Esteban Gutierrez created HBASE-15511: - Summary: ClusterStatus should be able Key: HBASE-15511 URL: https://issues.apache.org/jira/browse/HBASE-15511 Project: HBase Issue Type: Improvement Reporter: Esteban Gutierrez -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [NOTICE] defunct branch clean up
bq. the branch "hbase-6721" is not related to the actual patch(es) In that case, it is Okay to drop this branch. On Mon, Mar 21, 2016 at 11:51 AM, Sean Busbey wrote: > We should not accumulate feature branches indefinitely as it makes the repo > increasingly unwieldy. > > AFAICT, the branch "hbase-6721" is not related to the actual patch(es) that > went in to master for the HBASE-6721 implementation. It will not help with > an accurate backport. > > On Mon, Mar 21, 2016 at 1:22 PM, Ted Yu wrote: > > > For HBASE-6721, is there definitive decision not to backport to branch-1 > ? > > > > If not, keeping history for its branch is very helpful for backport. > > > > On Mon, Mar 21, 2016 at 11:14 AM, Enis Söztutar wrote: > > > > > What is our policy for deleting feature branches? For example, > > HBASE-10070, > > > HBASE-6721, etc are all committed, so we do not need the branches. We > > > should only keep them around if we think that keeping history is > > important > > > (which I am not so sure). > > > > > > Enis > > > > > > On Mon, Mar 21, 2016 at 11:03 AM, Sean Busbey > > wrote: > > > > > > > I'd like to clean out some of our defunct git branches. > > > > > > > > The current list I'm proposing is in JIRA: > > > > > > > > https://issues.apache.org/jira/browse/HBASE-15006 > > > > > > > > > > > > If you have a branch you're keeping around and haven't touched in > some > > > > time, please take a look and make sure I'm not including it. > > > > > > > > > > > > Similarly, if you know you had a feature branch at some point in the > > > > distant past and you're not interested in using it any more, please > > take > > > a > > > > look and make sure it's listed. > > > > -- > > > > busbey > > > > > > > > > > > > > -- > busbey >
[jira] [Created] (HBASE-15510) Reuse buffer inside RpcServer$Connection.readAndProcess
Vladimir Rodionov created HBASE-15510: - Summary: Reuse buffer inside RpcServer$Connection.readAndProcess Key: HBASE-15510 URL: https://issues.apache.org/jira/browse/HBASE-15510 Project: HBase Issue Type: Improvement Components: IPC/RPC Affects Versions: 2.0.0 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 2.0.0 ByteBuffer is created on every call. Server-side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15509) Avoid copy of block data in HFileBlock$Writer.finishBlock
Vladimir Rodionov created HBASE-15509: - Summary: Avoid copy of block data in HFileBlock$Writer.finishBlock Key: HBASE-15509 URL: https://issues.apache.org/jira/browse/HBASE-15509 Project: HBase Issue Type: Improvement Components: HFile Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov This calls ByteArrayOutputStream.toByteArray() which creates a copy of data block. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15508) Add command for exporting snapshot in hbase command script
Yufeng Jiang created HBASE-15508: Summary: Add command for exporting snapshot in hbase command script Key: HBASE-15508 URL: https://issues.apache.org/jira/browse/HBASE-15508 Project: HBase Issue Type: Improvement Components: hbase, scripts, snapshots Affects Versions: 1.1.3, 1.1.2, 1.2.0 Reporter: Yufeng Jiang Assignee: Yufeng Jiang Priority: Minor Fix For: 2.0.0, 1.2.1, 1.1.4, 1.2.2 Users of the hbase command script can now use the 'hbase snapshot export’ command. This replaces the need to type the full class name of 'org.apache.hadoop.hbase.snapshot.ExportSnapshot' In addition to adding command 'snapshot export', we are also grouping snapshot related commands together as subcommands under 'hbase snapshot'. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15507) Online modification of enabled ReplicationPeerConfig
Geoffrey Jacoby created HBASE-15507: --- Summary: Online modification of enabled ReplicationPeerConfig Key: HBASE-15507 URL: https://issues.apache.org/jira/browse/HBASE-15507 Project: HBase Issue Type: Improvement Components: Replication Affects Versions: 1.2.0, 2.0.0, 1.3.0 Reporter: Geoffrey Jacoby Assignee: Geoffrey Jacoby It's currently possible to update the table CFs for a replication peer while it's running, but not the peer configuration or data maps introduced as part of custom replication endpoints in HBASE-11367. This means that if you have a custom endpoint that depends on some configuration parameter, you have to remove the peer and recreate it in order to change the param. In some use cases that's not possible without risking data loss. HBASE-11393, which will consolidate tableCFs in the same znode as the rest of ReplicationPeerConfig, may help here, but with or without it it still seems like there needs to be further work to add update config/data API support to ReplicationAdmin and a callback mechanism to notify the endpoints of config parameter changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15506) FSDataOutputStream.write() allocates new byte buffer on each operation
Vladimir Rodionov created HBASE-15506: - Summary: FSDataOutputStream.write() allocates new byte buffer on each operation Key: HBASE-15506 URL: https://issues.apache.org/jira/browse/HBASE-15506 Project: HBase Issue Type: Improvement Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Deep inside stack trace in DFSOutputStream.createPacket. This should be opened in HDFS. This JIRA is to track HDFS work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15505) ReplicationPeerConfig should be builder-style
Enis Soztutar created HBASE-15505:
-
Summary: ReplicationPeerConfig should be builder-style
Key: HBASE-15505
URL: https://issues.apache.org/jira/browse/HBASE-15505
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Fix For: 2.0.0
Similar to HTD, HCD, {{ReplicationPeerConfig}} should be builder-style:
{code}
ReplicationPeerConfig().setFoo(foo).setBar(bar);
{code}
See {{TestHTableDescriptor.testClassMethodsAreBuilderStyle()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: [DISCUSS] Backporting Regionserver Groups (HBASE-6721) to 1.x and 0.98
bq. we instead focus on a releasable master +1 One of the hurdles to the above is that some unit tests in master are more flaky compared to their counterparts in branch-1. Smoothing out the flaky tests is non-trivial effort. On Mon, Mar 21, 2016 at 12:52 PM, Gary Helmling wrote: > > > > Tho based on this discussion there's a bunch of good features that users > > want that won't fit the criteria for #1 and #2. So allowing a backport of > > the few that can fit into the criteria shouldn't significantly affect > > future release from trunk. This way we can have some progress on some > > features. > > > > > If we have features that don't fit criteria #1 (compatibility), then we > need a new major release according to our semver guidelines. Criteria #2 > (stability) I think needs to be a gating factor on any new feature coming > in. We need to be able to demonstrate that we're not doing any harm for > existing users by adding new features. > > In both cases, I'm not sure we're very well served by committing new > features to master, then back-porting to branch-1. That means that > whatever stabilization is happening is likely from testing on branch-1 > based releases and not master, giving more time for changes to diverge. I > think we would be better served by releasing early and often from master > and not having a branch-1 at all. That would serve as a forcing function > to make sure that change going in to master have to stabilize while "fresh" > before a release is made. > > For more complex efforts, it would be better to stabilize in a feature > branch and then merge the completed effort as one. > > I think we're already boxing ourselves in by looking at 2.0 as a feature > based release, which means it necessarily has an unbounded timeline. That > means we have to do backports for other features ready before then, since > master can't be released. If we instead focus on a releasable master, that > should help us release large changes more incrementally, and save ourselves > the hassle of so many backports at the same time. Of course everyone is > free to invest their time where they choose. But I think if we focus on > time-based releases instead it would be better for the project as a whole. >
Re: Branch for 1.3
On Mon, Mar 21, 2016 at 2:05 PM, Mikhail Antonov wrote: > At this point branch-1.3 is very close (if different from at all) to > branch-1, so that's probably the same discussion. > > Fair enough. Is the previous discussion and documented gates (along with Andrew's assertion of users) sufficient? -- busbey
Re: [DISCUSS] Backporting Regionserver Groups (HBASE-6721) to 1.x and 0.98
> > Tho based on this discussion there's a bunch of good features that users > want that won't fit the criteria for #1 and #2. So allowing a backport of > the few that can fit into the criteria shouldn't significantly affect > future release from trunk. This way we can have some progress on some > features. > > If we have features that don't fit criteria #1 (compatibility), then we need a new major release according to our semver guidelines. Criteria #2 (stability) I think needs to be a gating factor on any new feature coming in. We need to be able to demonstrate that we're not doing any harm for existing users by adding new features. In both cases, I'm not sure we're very well served by committing new features to master, then back-porting to branch-1. That means that whatever stabilization is happening is likely from testing on branch-1 based releases and not master, giving more time for changes to diverge. I think we would be better served by releasing early and often from master and not having a branch-1 at all. That would serve as a forcing function to make sure that change going in to master have to stabilize while "fresh" before a release is made. For more complex efforts, it would be better to stabilize in a feature branch and then merge the completed effort as one. I think we're already boxing ourselves in by looking at 2.0 as a feature based release, which means it necessarily has an unbounded timeline. That means we have to do backports for other features ready before then, since master can't be released. If we instead focus on a releasable master, that should help us release large changes more incrementally, and save ourselves the hassle of so many backports at the same time. Of course everyone is free to invest their time where they choose. But I think if we focus on time-based releases instead it would be better for the project as a whole.
Re: Branch for 1.3
FWIW, I'd like to see the Spark connector get in. We have users who will be interested in it. On Mon, Mar 21, 2016 at 12:05 PM, Mikhail Antonov wrote: > At this point branch-1.3 is very close (if different from at all) to > branch-1, so that's probably the same discussion. > > -Mikhail > > On Mon, Mar 21, 2016 at 11:47 AM, Sean Busbey wrote: > > > Is the spark connector thread specifically about 1.3? or branch-1? > because > > we already had the branch-1 conversation. the specific gates were tracked > > in the umbrella jira. > > > > -Sean > > > > On Mon, Mar 21, 2016 at 1:33 PM, Mikhail Antonov > > wrote: > > > > > Yeah, we probably should start discussion thread about Spark connector. > > > Anyone wants to start the thread and push it forward? > > > > > > Regarding date-tiered compactions - since first impl already went in > 1.3, > > > would be good to get any possible improvements in 1.3 as well, as long > as > > > they are stable, IMO. > > > > > > -Mikhail > > > > > > On Mon, Mar 21, 2016 at 11:03 AM, Enis Söztutar > > > wrote: > > > > > > > You may want to track > > https://issues.apache.org/jira/browse/HBASE-15339 > > > as > > > > a parent for date-tiered compaction improvements. Current compaction > > > policy > > > > is useful in its own, but handling existing data, bulk loading etc > will > > > be > > > > improved with these subtasks. I think the patches can land before the > > 1.3 > > > > timeframe, but of course open for discussion for inclusion. My vote > > would > > > > be to include all the improvements since it would be easier to tell > the > > > > story to users. > > > > > > > > Enis > > > > > > > > On Mon, Mar 21, 2016 at 3:32 AM, Ted Yu wrote: > > > > > > > > > For Spark connector, we should start a separate discussion thread > > about > > > > > backporting to branch-1. > > > > > > > > > > Zhan has a bug fix coming this week which deals with how negative > > > numbers > > > > > are handled in comparison. > > > > > > > > > > FYI > > > > > > > > > > On Mon, Mar 21, 2016 at 1:35 AM, Mikhail Antonov < > > olorinb...@gmail.com > > > > > > > > > wrote: > > > > > > > > > > > Hi folks, > > > > > > > > > > > > bringing this topic up again. I'm planning to start spinning 1.3 > > > builds > > > > > and > > > > > > see if/where they break in a week or two, and (depending on how > it > > > > does) > > > > > > start preparing RCs in a month or maybe two. So, let's see where > we > > > > are. > > > > > > > > > > > > Big items first. There were long debates around three big items - > > > MOBs, > > > > > > Spark connector and RS groups, whether we should have them or > not. > > > > > > > > > > > > - MOBs > > > > > > I believe we decided that they aren't going to go in branch-1, > and > > > > hence > > > > > > not in branch-1.3 for sure. We might get back to that debate and > > > > > > re-consider MOBs for branch-1 if 2.0 is delayed, but almost > > certainly > > > > > they > > > > > > won't make it in 1.3 timeframe anyway as I feel. > > > > > > > > > > > > - Spark connector - HBASE-14160 it looks like it has 3 subtasks > > open, > > > > one > > > > > > of which is big one (HBASE-14375) - define public API for Spark > > > > > > integration. From the Jira looks like active work is happening on > > > other > > > > > > subtasks, but not on this one. So how's public API going? How > > stable > > > it > > > > > is? > > > > > > Who would want to have Spark in 1.3 and willing to help with > this? > > > > OTOH - > > > > > > who has objections about back-porting it? Has anyone been using > it > > in > > > > > some > > > > > > real environment? > > > > > > > > > > > > - RS groups - there was recently a thread about them, I'd like > to > > > > bring > > > > > it > > > > > > up again and get to some conclusion. > > > > > > > > > > > > Other features which we had in flight a month ago - > > > > > > > > > > > > - HBASE-15181 - date based tiered compactions has landed > > > > > > - HBASE-11290 - unlock RegionStates. I'm afraid the codebase has > > > moved > > > > > > forward quite a bit since the benchmark was run on this change > :( - > > > > > Francis > > > > > > - are you using it now? If we could have some benchmarks on the > > > latest > > > > > > rebase that I think would be great. > > > > > > > > > > > > > > > > > > - HBASE-13557 - special handling for system tables WALs - should > > we > > > > > still > > > > > > keep it targeted for 1.3? > > > > > > - HBASE-13017 - keep table state in meta, this one doesn't look > > like > > > > > it's > > > > > > going to make it in > > > > > > > > > > > > As a new item on my list - I'm looking forward to see more of > > > > HBASE-15492 > > > > > > (memory optimizations) subtasks to go in branch-1. > > > > > > > > > > > > Thanks! > > > > > > Mikhail > > > > > > > > > > > > On Sun, Feb 28, 2016 at 10:53 AM, Andrew Purtell < > > > > > andrew.purt...@gmail.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > I'm starting a push at work to get us up on 1.2. Ass
Re: Branch for 1.3
At this point branch-1.3 is very close (if different from at all) to branch-1, so that's probably the same discussion. -Mikhail On Mon, Mar 21, 2016 at 11:47 AM, Sean Busbey wrote: > Is the spark connector thread specifically about 1.3? or branch-1? because > we already had the branch-1 conversation. the specific gates were tracked > in the umbrella jira. > > -Sean > > On Mon, Mar 21, 2016 at 1:33 PM, Mikhail Antonov > wrote: > > > Yeah, we probably should start discussion thread about Spark connector. > > Anyone wants to start the thread and push it forward? > > > > Regarding date-tiered compactions - since first impl already went in 1.3, > > would be good to get any possible improvements in 1.3 as well, as long as > > they are stable, IMO. > > > > -Mikhail > > > > On Mon, Mar 21, 2016 at 11:03 AM, Enis Söztutar > > wrote: > > > > > You may want to track > https://issues.apache.org/jira/browse/HBASE-15339 > > as > > > a parent for date-tiered compaction improvements. Current compaction > > policy > > > is useful in its own, but handling existing data, bulk loading etc will > > be > > > improved with these subtasks. I think the patches can land before the > 1.3 > > > timeframe, but of course open for discussion for inclusion. My vote > would > > > be to include all the improvements since it would be easier to tell the > > > story to users. > > > > > > Enis > > > > > > On Mon, Mar 21, 2016 at 3:32 AM, Ted Yu wrote: > > > > > > > For Spark connector, we should start a separate discussion thread > about > > > > backporting to branch-1. > > > > > > > > Zhan has a bug fix coming this week which deals with how negative > > numbers > > > > are handled in comparison. > > > > > > > > FYI > > > > > > > > On Mon, Mar 21, 2016 at 1:35 AM, Mikhail Antonov < > olorinb...@gmail.com > > > > > > > wrote: > > > > > > > > > Hi folks, > > > > > > > > > > bringing this topic up again. I'm planning to start spinning 1.3 > > builds > > > > and > > > > > see if/where they break in a week or two, and (depending on how it > > > does) > > > > > start preparing RCs in a month or maybe two. So, let's see where we > > > are. > > > > > > > > > > Big items first. There were long debates around three big items - > > MOBs, > > > > > Spark connector and RS groups, whether we should have them or not. > > > > > > > > > > - MOBs > > > > > I believe we decided that they aren't going to go in branch-1, and > > > hence > > > > > not in branch-1.3 for sure. We might get back to that debate and > > > > > re-consider MOBs for branch-1 if 2.0 is delayed, but almost > certainly > > > > they > > > > > won't make it in 1.3 timeframe anyway as I feel. > > > > > > > > > > - Spark connector - HBASE-14160 it looks like it has 3 subtasks > open, > > > one > > > > > of which is big one (HBASE-14375) - define public API for Spark > > > > > integration. From the Jira looks like active work is happening on > > other > > > > > subtasks, but not on this one. So how's public API going? How > stable > > it > > > > is? > > > > > Who would want to have Spark in 1.3 and willing to help with this? > > > OTOH - > > > > > who has objections about back-porting it? Has anyone been using it > in > > > > some > > > > > real environment? > > > > > > > > > > - RS groups - there was recently a thread about them, I'd like to > > > bring > > > > it > > > > > up again and get to some conclusion. > > > > > > > > > > Other features which we had in flight a month ago - > > > > > > > > > > - HBASE-15181 - date based tiered compactions has landed > > > > > - HBASE-11290 - unlock RegionStates. I'm afraid the codebase has > > moved > > > > > forward quite a bit since the benchmark was run on this change :( - > > > > Francis > > > > > - are you using it now? If we could have some benchmarks on the > > latest > > > > > rebase that I think would be great. > > > > > > > > > > > > > > > - HBASE-13557 - special handling for system tables WALs - should > we > > > > still > > > > > keep it targeted for 1.3? > > > > > - HBASE-13017 - keep table state in meta, this one doesn't look > like > > > > it's > > > > > going to make it in > > > > > > > > > > As a new item on my list - I'm looking forward to see more of > > > HBASE-15492 > > > > > (memory optimizations) subtasks to go in branch-1. > > > > > > > > > > Thanks! > > > > > Mikhail > > > > > > > > > > On Sun, Feb 28, 2016 at 10:53 AM, Andrew Purtell < > > > > andrew.purt...@gmail.com > > > > > > > > > > > wrote: > > > > > > > > > > > I'm starting a push at work to get us up on 1.2. Assuming that > > > happens > > > > > > later this year I think that will be the end of my close > attention > > to > > > > > 0.98. > > > > > > > > > > > > > On Feb 26, 2016, at 1:54 PM, Nick Dimiduk > > > > wrote: > > > > > > > > > > > > > > On Fri, Feb 26, 2016 at 12:23 PM, Andrew Purtell < > > > > apurt...@apache.org> > > > > > > > wrote: > > > > > > > > > > > > > >> In the meantime those of us running HBase in production would > > > > benefit > > > >
Re: [NOTICE] defunct branch clean up
We should not accumulate feature branches indefinitely as it makes the repo increasingly unwieldy. AFAICT, the branch "hbase-6721" is not related to the actual patch(es) that went in to master for the HBASE-6721 implementation. It will not help with an accurate backport. On Mon, Mar 21, 2016 at 1:22 PM, Ted Yu wrote: > For HBASE-6721, is there definitive decision not to backport to branch-1 ? > > If not, keeping history for its branch is very helpful for backport. > > On Mon, Mar 21, 2016 at 11:14 AM, Enis Söztutar wrote: > > > What is our policy for deleting feature branches? For example, > HBASE-10070, > > HBASE-6721, etc are all committed, so we do not need the branches. We > > should only keep them around if we think that keeping history is > important > > (which I am not so sure). > > > > Enis > > > > On Mon, Mar 21, 2016 at 11:03 AM, Sean Busbey > wrote: > > > > > I'd like to clean out some of our defunct git branches. > > > > > > The current list I'm proposing is in JIRA: > > > > > > https://issues.apache.org/jira/browse/HBASE-15006 > > > > > > > > > If you have a branch you're keeping around and haven't touched in some > > > time, please take a look and make sure I'm not including it. > > > > > > > > > Similarly, if you know you had a feature branch at some point in the > > > distant past and you're not interested in using it any more, please > take > > a > > > look and make sure it's listed. > > > -- > > > busbey > > > > > > -- busbey
[jira] [Created] (HBASE-15504) Fix Balancer in 1.3 not moving regions off overloaded regionserver
Elliott Clark created HBASE-15504: - Summary: Fix Balancer in 1.3 not moving regions off overloaded regionserver Key: HBASE-15504 URL: https://issues.apache.org/jira/browse/HBASE-15504 Project: HBase Issue Type: Bug Reporter: Elliott Clark We pushed 1.3 to a couple of clusters. In some cases the regions were assigned VERY un-evenly and the regions would not move after that. We ended up with one rs getting thousands of regions and most servers getting 0. Running balancer would do nothing. The balancer would say that it couldn't find a solution with less than the current cost. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-15502) Skeleton unit test to copy/paste
[ https://issues.apache.org/jira/browse/HBASE-15502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-15502. --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.0.0 Thanks [~misty] Pushed...to master > Skeleton unit test to copy/paste > > > Key: HBASE-15502 > URL: https://issues.apache.org/jira/browse/HBASE-15502 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: 15502.patch > > > Add to our docs a skeleton unit test to copy/paste with all the vitals > already filled out such as category and categorybasetimeout @Rule. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Branch for 1.3
Is the spark connector thread specifically about 1.3? or branch-1? because we already had the branch-1 conversation. the specific gates were tracked in the umbrella jira. -Sean On Mon, Mar 21, 2016 at 1:33 PM, Mikhail Antonov wrote: > Yeah, we probably should start discussion thread about Spark connector. > Anyone wants to start the thread and push it forward? > > Regarding date-tiered compactions - since first impl already went in 1.3, > would be good to get any possible improvements in 1.3 as well, as long as > they are stable, IMO. > > -Mikhail > > On Mon, Mar 21, 2016 at 11:03 AM, Enis Söztutar > wrote: > > > You may want to track https://issues.apache.org/jira/browse/HBASE-15339 > as > > a parent for date-tiered compaction improvements. Current compaction > policy > > is useful in its own, but handling existing data, bulk loading etc will > be > > improved with these subtasks. I think the patches can land before the 1.3 > > timeframe, but of course open for discussion for inclusion. My vote would > > be to include all the improvements since it would be easier to tell the > > story to users. > > > > Enis > > > > On Mon, Mar 21, 2016 at 3:32 AM, Ted Yu wrote: > > > > > For Spark connector, we should start a separate discussion thread about > > > backporting to branch-1. > > > > > > Zhan has a bug fix coming this week which deals with how negative > numbers > > > are handled in comparison. > > > > > > FYI > > > > > > On Mon, Mar 21, 2016 at 1:35 AM, Mikhail Antonov > > > > wrote: > > > > > > > Hi folks, > > > > > > > > bringing this topic up again. I'm planning to start spinning 1.3 > builds > > > and > > > > see if/where they break in a week or two, and (depending on how it > > does) > > > > start preparing RCs in a month or maybe two. So, let's see where we > > are. > > > > > > > > Big items first. There were long debates around three big items - > MOBs, > > > > Spark connector and RS groups, whether we should have them or not. > > > > > > > > - MOBs > > > > I believe we decided that they aren't going to go in branch-1, and > > hence > > > > not in branch-1.3 for sure. We might get back to that debate and > > > > re-consider MOBs for branch-1 if 2.0 is delayed, but almost certainly > > > they > > > > won't make it in 1.3 timeframe anyway as I feel. > > > > > > > > - Spark connector - HBASE-14160 it looks like it has 3 subtasks open, > > one > > > > of which is big one (HBASE-14375) - define public API for Spark > > > > integration. From the Jira looks like active work is happening on > other > > > > subtasks, but not on this one. So how's public API going? How stable > it > > > is? > > > > Who would want to have Spark in 1.3 and willing to help with this? > > OTOH - > > > > who has objections about back-porting it? Has anyone been using it in > > > some > > > > real environment? > > > > > > > > - RS groups - there was recently a thread about them, I'd like to > > bring > > > it > > > > up again and get to some conclusion. > > > > > > > > Other features which we had in flight a month ago - > > > > > > > > - HBASE-15181 - date based tiered compactions has landed > > > > - HBASE-11290 - unlock RegionStates. I'm afraid the codebase has > moved > > > > forward quite a bit since the benchmark was run on this change :( - > > > Francis > > > > - are you using it now? If we could have some benchmarks on the > latest > > > > rebase that I think would be great. > > > > > > > > > > > > - HBASE-13557 - special handling for system tables WALs - should we > > > still > > > > keep it targeted for 1.3? > > > > - HBASE-13017 - keep table state in meta, this one doesn't look like > > > it's > > > > going to make it in > > > > > > > > As a new item on my list - I'm looking forward to see more of > > HBASE-15492 > > > > (memory optimizations) subtasks to go in branch-1. > > > > > > > > Thanks! > > > > Mikhail > > > > > > > > On Sun, Feb 28, 2016 at 10:53 AM, Andrew Purtell < > > > andrew.purt...@gmail.com > > > > > > > > > wrote: > > > > > > > > > I'm starting a push at work to get us up on 1.2. Assuming that > > happens > > > > > later this year I think that will be the end of my close attention > to > > > > 0.98. > > > > > > > > > > > On Feb 26, 2016, at 1:54 PM, Nick Dimiduk > > > wrote: > > > > > > > > > > > > On Fri, Feb 26, 2016 at 12:23 PM, Andrew Purtell < > > > apurt...@apache.org> > > > > > > wrote: > > > > > > > > > > > >> In the meantime those of us running HBase in production would > > > benefit > > > > > from > > > > > >> fairly frequent minor releases. > > > > > > > > > > > > +1. Having to look back to 0.98 to get some new feature is > > > problematic. > > > > > > > > > > > >> On Fri, Feb 26, 2016 at 9:50 AM, Elliott Clark < > ecl...@apache.org > > > > > > > > wrote: > > > > > >> > > > > > >>> I disagree. We have agreed that 2.0 will have a new assignement > > > > > manager. > > > > > >>> There's a lot of work that has been done on getting that in, so > > far > > > > > there > > > > > >>> are n
Re: Branch for 1.3
Yeah, we probably should start discussion thread about Spark connector. Anyone wants to start the thread and push it forward? Regarding date-tiered compactions - since first impl already went in 1.3, would be good to get any possible improvements in 1.3 as well, as long as they are stable, IMO. -Mikhail On Mon, Mar 21, 2016 at 11:03 AM, Enis Söztutar wrote: > You may want to track https://issues.apache.org/jira/browse/HBASE-15339 as > a parent for date-tiered compaction improvements. Current compaction policy > is useful in its own, but handling existing data, bulk loading etc will be > improved with these subtasks. I think the patches can land before the 1.3 > timeframe, but of course open for discussion for inclusion. My vote would > be to include all the improvements since it would be easier to tell the > story to users. > > Enis > > On Mon, Mar 21, 2016 at 3:32 AM, Ted Yu wrote: > > > For Spark connector, we should start a separate discussion thread about > > backporting to branch-1. > > > > Zhan has a bug fix coming this week which deals with how negative numbers > > are handled in comparison. > > > > FYI > > > > On Mon, Mar 21, 2016 at 1:35 AM, Mikhail Antonov > > wrote: > > > > > Hi folks, > > > > > > bringing this topic up again. I'm planning to start spinning 1.3 builds > > and > > > see if/where they break in a week or two, and (depending on how it > does) > > > start preparing RCs in a month or maybe two. So, let's see where we > are. > > > > > > Big items first. There were long debates around three big items - MOBs, > > > Spark connector and RS groups, whether we should have them or not. > > > > > > - MOBs > > > I believe we decided that they aren't going to go in branch-1, and > hence > > > not in branch-1.3 for sure. We might get back to that debate and > > > re-consider MOBs for branch-1 if 2.0 is delayed, but almost certainly > > they > > > won't make it in 1.3 timeframe anyway as I feel. > > > > > > - Spark connector - HBASE-14160 it looks like it has 3 subtasks open, > one > > > of which is big one (HBASE-14375) - define public API for Spark > > > integration. From the Jira looks like active work is happening on other > > > subtasks, but not on this one. So how's public API going? How stable it > > is? > > > Who would want to have Spark in 1.3 and willing to help with this? > OTOH - > > > who has objections about back-porting it? Has anyone been using it in > > some > > > real environment? > > > > > > - RS groups - there was recently a thread about them, I'd like to > bring > > it > > > up again and get to some conclusion. > > > > > > Other features which we had in flight a month ago - > > > > > > - HBASE-15181 - date based tiered compactions has landed > > > - HBASE-11290 - unlock RegionStates. I'm afraid the codebase has moved > > > forward quite a bit since the benchmark was run on this change :( - > > Francis > > > - are you using it now? If we could have some benchmarks on the latest > > > rebase that I think would be great. > > > > > > > > > - HBASE-13557 - special handling for system tables WALs - should we > > still > > > keep it targeted for 1.3? > > > - HBASE-13017 - keep table state in meta, this one doesn't look like > > it's > > > going to make it in > > > > > > As a new item on my list - I'm looking forward to see more of > HBASE-15492 > > > (memory optimizations) subtasks to go in branch-1. > > > > > > Thanks! > > > Mikhail > > > > > > On Sun, Feb 28, 2016 at 10:53 AM, Andrew Purtell < > > andrew.purt...@gmail.com > > > > > > > wrote: > > > > > > > I'm starting a push at work to get us up on 1.2. Assuming that > happens > > > > later this year I think that will be the end of my close attention to > > > 0.98. > > > > > > > > > On Feb 26, 2016, at 1:54 PM, Nick Dimiduk > > wrote: > > > > > > > > > > On Fri, Feb 26, 2016 at 12:23 PM, Andrew Purtell < > > apurt...@apache.org> > > > > > wrote: > > > > > > > > > >> In the meantime those of us running HBase in production would > > benefit > > > > from > > > > >> fairly frequent minor releases. > > > > > > > > > > +1. Having to look back to 0.98 to get some new feature is > > problematic. > > > > > > > > > >> On Fri, Feb 26, 2016 at 9:50 AM, Elliott Clark > > > > > wrote: > > > > >> > > > > >>> I disagree. We have agreed that 2.0 will have a new assignement > > > > manager. > > > > >>> There's a lot of work that has been done on getting that in, so > far > > > > there > > > > >>> are no benefits to the end user from all that work. We should > stick > > > > with > > > > >>> the plan and release 2.0 when it's ready. > > > > >>> > > > > >>> On Fri, Feb 26, 2016 at 9:39 AM, Stephen Jiang < > > > > syuanjiang...@gmail.com> > > > > >>> wrote: > > > > >>> > > > > Thanks for Mikhail for taking the 1.3 RM role. Looks like we > > have a > > > > >> lot > > > > >>> of > > > > new things in 1.3 release. > > > > > > > > Based on the experience of 1.1 and 1.2 release, it takes a lot > of > > > > >> efforts > >
[jira] [Created] (HBASE-15503) Document REST endpoints for namespaces
Misty Stanley-Jones created HBASE-15503: --- Summary: Document REST endpoints for namespaces Key: HBASE-15503 URL: https://issues.apache.org/jira/browse/HBASE-15503 Project: HBase Issue Type: Sub-task Components: documentation Reporter: Misty Stanley-Jones See HBASE-14147 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] Backporting Regionserver Groups (HBASE-6721) to 1.x and 0.98
I see, that is indeed undesirable. Tho based on this discussion there's a bunch of good features that users want that won't fit the criteria for #1 and #2. So allowing a backport of the few that can fit into the criteria shouldn't significantly affect future release from trunk. This way we can have some progress on some features. On Monday, March 21, 2016 10:23 AM, Sean Busbey wrote: Hadoop's trunk branch hasn't had a release in a very very long time. So it continues to accumulate changes that aren't in a release while folks drive their particular desired features back into branch-2. On Mon, Mar 21, 2016 at 12:01 PM, Francis Liu wrote: To summarize so far it seems the concerns for backporting are: 1. compatibility - api, wire, rolling upgradeabability2. stability - destabilizing code and deploys for those that don't want the new feature Is there anything else? Elliot what happened to hadoop? Is it neither of the two? Francis On Thursday, March 17, 2016 12:01 PM, Enis Söztutar wrote: As long as there is interest in doing the backport work, maintaining and experimenting with the feature on an actual release (which seems to be the case for RSGroups and Spark at least) and keeping the code base for branch-1 stable, there is no reason not to backport. RsGroups (and I believe Spark support as well) is marked experimental in release notes which means that there is some room for our client-visible guarantees. 2.0 has it's own timeline and feature set and will come in time. Of course anybody can propose to cut a release and push for getting it sooner rather than later. We should all help the effort, but realistically, even if after 2.0 is released, there will be people running 1.x for years after that. Enis On Thu, Mar 17, 2016 at 11:28 AM, Mikhail Antonov wrote: > >>>"Why MOB and RegionServer Groups should be in a new major version and > stuff > like the new RPC queue (HBASE-15136), date based tiered compactions > (HBASE-15181), special handling for system tables WALs (HBASE-13557), keep > table state in meta (HBASE-13017) or the Region Normalizer (HBASE-13103) > are considered for or already in 1.x?" > > To me the differentiator would be "how much does it change the codebase > around". If all/most of code change is the code which is new/ignored when > feature is turned off, and by default it's off until well-tested by various > users - then it should be fine to include. In the list above MOBs probably > don't satisfy that, Spark and RS Groups probably do. > > If we make 2.0 release with just Mobs and RS Groups, that would mean new AM > would have to be postponed to 3.0? What about procsV2? Although we would > want rolling upgrade to 2.0, still it's our chance to release something > big, invasive and new (since as was mentioned, the user expectation anyway > would be that in new major version some incompatibilities would be present > and some migration may be required)? > > -Mikhail > > On Thu, Mar 17, 2016 at 7:48 AM, Matteo Bertozzi > wrote: > > > Why MOB and RegionServer Groups should be in a new major version and > stuff > > like the new RPC queue (HBASE-15136), date based tiered compactions > > (HBASE-15181), special handling for system tables WALs (HBASE-13557), > keep > > table state in meta (HBASE-13017) or the Region Normalizer (HBASE-13103) > > are considered for or already in 1.x? > > > > to me, and probably most of the users, a new Major version means that > APIs > > will break, wire may break, there may be an upgrade of some sort and so > on. > > which is not true for MOB and RS groups. > > > > In case we do a major for RS groups and Mob will that still based on the > > 1.x branch? > > > > On Wed, Mar 16, 2016 at 11:23 PM, Andrew Purtell < > andrew.purt...@gmail.com > > > > > wrote: > > > > > I remember explicitly saying I was not against a backport of the MOB > > > feature. You are misrepresenting my position a bit. Sure, I'm a > skeptic. > > > Not a big deal because I don't think we can or should seek a blanket > > rule. > > > Sometimes a feature will have sufficient interest and applicability > that > > a > > > backport is worth considering, and then there's a question of what kind > > of > > > risk the changes envisioned carry. RS groups as implemented are low > risk. > > > Meanwhile MOB is highly invasive. I think RS groups would have two > large > > > production users soon after introduction into branch-1. I'm not sure > > about > > > MOB. They are worth consideration on their own merits and on user > demand > > > for them. > > > > > > Another thing we could do is get 2.0 started right now. Whatever major > > > that doesn't make the cut could go into 3.0. I think the requests for > > these > > > backports are coming because there is no near time horizon for a 2.0 > > > release. So set it soon? > > > > > > > > > > On Mar 16, 2016, at 9:27 PM, Elliott Clark > wrote: > > > > > > > > On Wed, Mar 16, 2016 at 6:26 PM, Andrew Purtell < > > > andrew.purt...@gm
[jira] [Created] (HBASE-15502) Skeleton unit test to copy/paste
stack created HBASE-15502: - Summary: Skeleton unit test to copy/paste Key: HBASE-15502 URL: https://issues.apache.org/jira/browse/HBASE-15502 Project: HBase Issue Type: Task Components: documentation Reporter: stack Assignee: stack Add to our docs a skeleton unit test to copy/paste with all the vitals already filled out such as category and categorybasetimeout @Rule. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15501) MultiAction.add creates unnecessary temporary objects
Vladimir Rodionov created HBASE-15501:
-
Summary: MultiAction.add creates unnecessary temporary objects
Key: HBASE-15501
URL: https://issues.apache.org/jira/browse/HBASE-15501
Project: HBase
Issue Type: Improvement
Components: Client
Affects Versions: 2.0.0
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov
Fix For: 2.0.0
MultiAction class:
{code}
/**
* Add an Action to this container based on it's regionName. If the regionName
* is wrong, the initial execution will fail, but will be automatically
* retried after looking up the correct region.
*
* @param regionName
* @param a
*/
public void add(byte[] regionName, Action a) {
add(regionName, Arrays.asList(a));
}
/**
* Add an Action to this container based on it's regionName. If the regionName
* is wrong, the initial execution will fail, but will be automatically
* retried after looking up the correct region.
*
* @param regionName
* @param actionList list of actions to add for the region
*/
public void add(byte[] regionName, List> actionList){
List> rsActions = actions.get(regionName);
if (rsActions == null) {
rsActions = new ArrayList>(actionList.size());
actions.put(regionName, rsActions);
}
rsActions.addAll(actionList);
}
{code}
Avoid Arrays.asList(a) and Collection.addAll - they create temporary garbage
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: [NOTICE] defunct branch clean up
For HBASE-6721, is there definitive decision not to backport to branch-1 ? If not, keeping history for its branch is very helpful for backport. On Mon, Mar 21, 2016 at 11:14 AM, Enis Söztutar wrote: > What is our policy for deleting feature branches? For example, HBASE-10070, > HBASE-6721, etc are all committed, so we do not need the branches. We > should only keep them around if we think that keeping history is important > (which I am not so sure). > > Enis > > On Mon, Mar 21, 2016 at 11:03 AM, Sean Busbey wrote: > > > I'd like to clean out some of our defunct git branches. > > > > The current list I'm proposing is in JIRA: > > > > https://issues.apache.org/jira/browse/HBASE-15006 > > > > > > If you have a branch you're keeping around and haven't touched in some > > time, please take a look and make sure I'm not including it. > > > > > > Similarly, if you know you had a feature branch at some point in the > > distant past and you're not interested in using it any more, please take > a > > look and make sure it's listed. > > -- > > busbey > > >
Re: [NOTICE] defunct branch clean up
What is our policy for deleting feature branches? For example, HBASE-10070, HBASE-6721, etc are all committed, so we do not need the branches. We should only keep them around if we think that keeping history is important (which I am not so sure). Enis On Mon, Mar 21, 2016 at 11:03 AM, Sean Busbey wrote: > I'd like to clean out some of our defunct git branches. > > The current list I'm proposing is in JIRA: > > https://issues.apache.org/jira/browse/HBASE-15006 > > > If you have a branch you're keeping around and haven't touched in some > time, please take a look and make sure I'm not including it. > > > Similarly, if you know you had a feature branch at some point in the > distant past and you're not interested in using it any more, please take a > look and make sure it's listed. > -- > busbey >
Re: Branch for 1.3
You may want to track https://issues.apache.org/jira/browse/HBASE-15339 as a parent for date-tiered compaction improvements. Current compaction policy is useful in its own, but handling existing data, bulk loading etc will be improved with these subtasks. I think the patches can land before the 1.3 timeframe, but of course open for discussion for inclusion. My vote would be to include all the improvements since it would be easier to tell the story to users. Enis On Mon, Mar 21, 2016 at 3:32 AM, Ted Yu wrote: > For Spark connector, we should start a separate discussion thread about > backporting to branch-1. > > Zhan has a bug fix coming this week which deals with how negative numbers > are handled in comparison. > > FYI > > On Mon, Mar 21, 2016 at 1:35 AM, Mikhail Antonov > wrote: > > > Hi folks, > > > > bringing this topic up again. I'm planning to start spinning 1.3 builds > and > > see if/where they break in a week or two, and (depending on how it does) > > start preparing RCs in a month or maybe two. So, let's see where we are. > > > > Big items first. There were long debates around three big items - MOBs, > > Spark connector and RS groups, whether we should have them or not. > > > > - MOBs > > I believe we decided that they aren't going to go in branch-1, and hence > > not in branch-1.3 for sure. We might get back to that debate and > > re-consider MOBs for branch-1 if 2.0 is delayed, but almost certainly > they > > won't make it in 1.3 timeframe anyway as I feel. > > > > - Spark connector - HBASE-14160 it looks like it has 3 subtasks open, one > > of which is big one (HBASE-14375) - define public API for Spark > > integration. From the Jira looks like active work is happening on other > > subtasks, but not on this one. So how's public API going? How stable it > is? > > Who would want to have Spark in 1.3 and willing to help with this? OTOH - > > who has objections about back-porting it? Has anyone been using it in > some > > real environment? > > > > - RS groups - there was recently a thread about them, I'd like to bring > it > > up again and get to some conclusion. > > > > Other features which we had in flight a month ago - > > > > - HBASE-15181 - date based tiered compactions has landed > > - HBASE-11290 - unlock RegionStates. I'm afraid the codebase has moved > > forward quite a bit since the benchmark was run on this change :( - > Francis > > - are you using it now? If we could have some benchmarks on the latest > > rebase that I think would be great. > > > > > > - HBASE-13557 - special handling for system tables WALs - should we > still > > keep it targeted for 1.3? > > - HBASE-13017 - keep table state in meta, this one doesn't look like > it's > > going to make it in > > > > As a new item on my list - I'm looking forward to see more of HBASE-15492 > > (memory optimizations) subtasks to go in branch-1. > > > > Thanks! > > Mikhail > > > > On Sun, Feb 28, 2016 at 10:53 AM, Andrew Purtell < > andrew.purt...@gmail.com > > > > > wrote: > > > > > I'm starting a push at work to get us up on 1.2. Assuming that happens > > > later this year I think that will be the end of my close attention to > > 0.98. > > > > > > > On Feb 26, 2016, at 1:54 PM, Nick Dimiduk > wrote: > > > > > > > > On Fri, Feb 26, 2016 at 12:23 PM, Andrew Purtell < > apurt...@apache.org> > > > > wrote: > > > > > > > >> In the meantime those of us running HBase in production would > benefit > > > from > > > >> fairly frequent minor releases. > > > > > > > > +1. Having to look back to 0.98 to get some new feature is > problematic. > > > > > > > >> On Fri, Feb 26, 2016 at 9:50 AM, Elliott Clark > > > wrote: > > > >> > > > >>> I disagree. We have agreed that 2.0 will have a new assignement > > > manager. > > > >>> There's a lot of work that has been done on getting that in, so far > > > there > > > >>> are no benefits to the end user from all that work. We should stick > > > with > > > >>> the plan and release 2.0 when it's ready. > > > >>> > > > >>> On Fri, Feb 26, 2016 at 9:39 AM, Stephen Jiang < > > > syuanjiang...@gmail.com> > > > >>> wrote: > > > >>> > > > Thanks for Mikhail for taking the 1.3 RM role. Looks like we > have a > > > >> lot > > > >>> of > > > new things in 1.3 release. > > > > > > Based on the experience of 1.1 and 1.2 release, it takes a lot of > > > >> efforts > > > to get a stable minor release out. From this, I have my own > 2-cents > > > on > > > >>> 1.4 > > > release. The plan is to have 2.0 release during summer time of > this > > > >> year > > > (yeah, *this year). * Given the limited time and resource, after > > 1.3 > > > release, instead of spending effort on 1.4 release, the community > > > >> should > > > focus on stabilizing master (or branch-2, not exist as of now) > > branch > > > >> and > > > make 2.0 release a priority. 2.0 release would bring more values > to > > > customer & move towards maturity of HBASE product. > > > > >
[NOTICE] defunct branch clean up
I'd like to clean out some of our defunct git branches. The current list I'm proposing is in JIRA: https://issues.apache.org/jira/browse/HBASE-15006 If you have a branch you're keeping around and haven't touched in some time, please take a look and make sure I'm not including it. Similarly, if you know you had a feature branch at some point in the distant past and you're not interested in using it any more, please take a look and make sure it's listed. -- busbey
Re: [DISCUSS] Backporting Regionserver Groups (HBASE-6721) to 1.x and 0.98
Hadoop's trunk branch hasn't had a release in a very very long time. So it continues to accumulate changes that aren't in a release while folks drive their particular desired features back into branch-2. On Mon, Mar 21, 2016 at 12:01 PM, Francis Liu wrote: > To summarize so far it seems the concerns for backporting are: > 1. compatibility - api, wire, rolling upgradeabability2. stability - > destabilizing code and deploys for those that don't want the new feature > Is there anything else? > Elliot what happened to hadoop? Is it neither of the two? > Francis > > > > > On Thursday, March 17, 2016 12:01 PM, Enis Söztutar > wrote: > > > As long as there is interest in doing the backport work, maintaining and > experimenting with the feature on an actual release (which seems to be the > case for RSGroups and Spark at least) and keeping the code base for > branch-1 stable, there is no reason not to backport. RsGroups (and I > believe Spark support as well) is marked experimental in release notes > which means that there is some room for our client-visible guarantees. > > 2.0 has it's own timeline and feature set and will come in time. Of course > anybody can propose to cut a release and push for getting it sooner rather > than later. We should all help the effort, but realistically, even if after > 2.0 is released, there will be people running 1.x for years after that. > > Enis > > On Thu, Mar 17, 2016 at 11:28 AM, Mikhail Antonov > wrote: > > > >>>"Why MOB and RegionServer Groups should be in a new major version and > > stuff > > like the new RPC queue (HBASE-15136), date based tiered compactions > > (HBASE-15181), special handling for system tables WALs (HBASE-13557), > keep > > table state in meta (HBASE-13017) or the Region Normalizer (HBASE-13103) > > are considered for or already in 1.x?" > > > > To me the differentiator would be "how much does it change the codebase > > around". If all/most of code change is the code which is new/ignored when > > feature is turned off, and by default it's off until well-tested by > various > > users - then it should be fine to include. In the list above MOBs > probably > > don't satisfy that, Spark and RS Groups probably do. > > > > If we make 2.0 release with just Mobs and RS Groups, that would mean new > AM > > would have to be postponed to 3.0? What about procsV2? Although we would > > want rolling upgrade to 2.0, still it's our chance to release something > > big, invasive and new (since as was mentioned, the user expectation > anyway > > would be that in new major version some incompatibilities would be > present > > and some migration may be required)? > > > > -Mikhail > > > > On Thu, Mar 17, 2016 at 7:48 AM, Matteo Bertozzi < > theo.berto...@gmail.com> > > wrote: > > > > > Why MOB and RegionServer Groups should be in a new major version and > > stuff > > > like the new RPC queue (HBASE-15136), date based tiered compactions > > > (HBASE-15181), special handling for system tables WALs (HBASE-13557), > > keep > > > table state in meta (HBASE-13017) or the Region Normalizer > (HBASE-13103) > > > are considered for or already in 1.x? > > > > > > to me, and probably most of the users, a new Major version means that > > APIs > > > will break, wire may break, there may be an upgrade of some sort and so > > on. > > > which is not true for MOB and RS groups. > > > > > > In case we do a major for RS groups and Mob will that still based on > the > > > 1.x branch? > > > > > > On Wed, Mar 16, 2016 at 11:23 PM, Andrew Purtell < > > andrew.purt...@gmail.com > > > > > > > wrote: > > > > > > > I remember explicitly saying I was not against a backport of the MOB > > > > feature. You are misrepresenting my position a bit. Sure, I'm a > > skeptic. > > > > Not a big deal because I don't think we can or should seek a blanket > > > rule. > > > > Sometimes a feature will have sufficient interest and applicability > > that > > > a > > > > backport is worth considering, and then there's a question of what > kind > > > of > > > > risk the changes envisioned carry. RS groups as implemented are low > > risk. > > > > Meanwhile MOB is highly invasive. I think RS groups would have two > > large > > > > production users soon after introduction into branch-1. I'm not sure > > > about > > > > MOB. They are worth consideration on their own merits and on user > > demand > > > > for them. > > > > > > > > Another thing we could do is get 2.0 started right now. Whatever > major > > > > that doesn't make the cut could go into 3.0. I think the requests for > > > these > > > > backports are coming because there is no near time horizon for a 2.0 > > > > release. So set it soon? > > > > > > > > > > > > > On Mar 16, 2016, at 9:27 PM, Elliott Clark > > wrote: > > > > > > > > > > On Wed, Mar 16, 2016 at 6:26 PM, Andrew Purtell < > > > > andrew.purt...@gmail.com> > > > > > wrote: > > > > > > > > > >> Because I for one might well want to run RS groups in production > > with > > > > >> branch-1 code witho
Re: [DISCUSS] Backporting Regionserver Groups (HBASE-6721) to 1.x and 0.98
To summarize so far it seems the concerns for backporting are: 1. compatibility - api, wire, rolling upgradeabability2. stability - destabilizing code and deploys for those that don't want the new feature Is there anything else? Elliot what happened to hadoop? Is it neither of the two? Francis On Thursday, March 17, 2016 12:01 PM, Enis Söztutar wrote: As long as there is interest in doing the backport work, maintaining and experimenting with the feature on an actual release (which seems to be the case for RSGroups and Spark at least) and keeping the code base for branch-1 stable, there is no reason not to backport. RsGroups (and I believe Spark support as well) is marked experimental in release notes which means that there is some room for our client-visible guarantees. 2.0 has it's own timeline and feature set and will come in time. Of course anybody can propose to cut a release and push for getting it sooner rather than later. We should all help the effort, but realistically, even if after 2.0 is released, there will be people running 1.x for years after that. Enis On Thu, Mar 17, 2016 at 11:28 AM, Mikhail Antonov wrote: > >>>"Why MOB and RegionServer Groups should be in a new major version and > stuff > like the new RPC queue (HBASE-15136), date based tiered compactions > (HBASE-15181), special handling for system tables WALs (HBASE-13557), keep > table state in meta (HBASE-13017) or the Region Normalizer (HBASE-13103) > are considered for or already in 1.x?" > > To me the differentiator would be "how much does it change the codebase > around". If all/most of code change is the code which is new/ignored when > feature is turned off, and by default it's off until well-tested by various > users - then it should be fine to include. In the list above MOBs probably > don't satisfy that, Spark and RS Groups probably do. > > If we make 2.0 release with just Mobs and RS Groups, that would mean new AM > would have to be postponed to 3.0? What about procsV2? Although we would > want rolling upgrade to 2.0, still it's our chance to release something > big, invasive and new (since as was mentioned, the user expectation anyway > would be that in new major version some incompatibilities would be present > and some migration may be required)? > > -Mikhail > > On Thu, Mar 17, 2016 at 7:48 AM, Matteo Bertozzi > wrote: > > > Why MOB and RegionServer Groups should be in a new major version and > stuff > > like the new RPC queue (HBASE-15136), date based tiered compactions > > (HBASE-15181), special handling for system tables WALs (HBASE-13557), > keep > > table state in meta (HBASE-13017) or the Region Normalizer (HBASE-13103) > > are considered for or already in 1.x? > > > > to me, and probably most of the users, a new Major version means that > APIs > > will break, wire may break, there may be an upgrade of some sort and so > on. > > which is not true for MOB and RS groups. > > > > In case we do a major for RS groups and Mob will that still based on the > > 1.x branch? > > > > On Wed, Mar 16, 2016 at 11:23 PM, Andrew Purtell < > andrew.purt...@gmail.com > > > > > wrote: > > > > > I remember explicitly saying I was not against a backport of the MOB > > > feature. You are misrepresenting my position a bit. Sure, I'm a > skeptic. > > > Not a big deal because I don't think we can or should seek a blanket > > rule. > > > Sometimes a feature will have sufficient interest and applicability > that > > a > > > backport is worth considering, and then there's a question of what kind > > of > > > risk the changes envisioned carry. RS groups as implemented are low > risk. > > > Meanwhile MOB is highly invasive. I think RS groups would have two > large > > > production users soon after introduction into branch-1. I'm not sure > > about > > > MOB. They are worth consideration on their own merits and on user > demand > > > for them. > > > > > > Another thing we could do is get 2.0 started right now. Whatever major > > > that doesn't make the cut could go into 3.0. I think the requests for > > these > > > backports are coming because there is no near time horizon for a 2.0 > > > release. So set it soon? > > > > > > > > > > On Mar 16, 2016, at 9:27 PM, Elliott Clark > wrote: > > > > > > > > On Wed, Mar 16, 2016 at 6:26 PM, Andrew Purtell < > > > andrew.purt...@gmail.com> > > > > wrote: > > > > > > > >> Because I for one might well want to run RS groups in production > with > > > >> branch-1 code without waiting for or dealing with the other stuff > > > coming in > > > >> any 2.0. > > > > > > > > > > > > This is the same email that I sent for MOB. Which you agreed with > then. > > > But > > > > not now. There's nothing substantively different about this feature > > that > > > > makes it different from any other feature. It's a large change that > > > wasn't > > > > there in 1.X line. > > > > > > > > I would like backups, and procedure v2 in 1.x. However even if it > > landed > > > > tomorrow they shouldn't be back p
[jira] [Created] (HBASE-15500) Cut HBase 1.2.1 release
Sean Busbey created HBASE-15500: --- Summary: Cut HBase 1.2.1 release Key: HBASE-15500 URL: https://issues.apache.org/jira/browse/HBASE-15500 Project: HBase Issue Type: Task Components: community Reporter: Sean Busbey Assignee: Sean Busbey Fix For: 1.2.1 we're due for 1.2.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Successful: HBase Generate Website
Build status: Successful
If successful, the website and docs have been generated. If failed, skip to the
bottom of this email.
Use the following commands to download the patch and apply it to a clean branch
based on origin/asf-site. If you prefer to keep the hbase-site repo around
permanently, you can skip the clone step.
git clone https://git-wip-us.apache.org/repos/asf/hbase-site.git
cd hbase-site
wget -O-
https://builds.apache.org/job/hbase_generate_website/173/artifact/website.patch.zip
| funzip > f1d453599a90b8a1075082e2dc7b7676416f.patch
git fetch
git checkout -b asf-site-f1d453599a90b8a1075082e2dc7b7676416f
origin/asf-site
git am f1d453599a90b8a1075082e2dc7b7676416f.patch
At this point, you can preview the changes by opening index.html or any of the
other HTML pages in your local
asf-site-f1d453599a90b8a1075082e2dc7b7676416f branch, and you can review
the differences by running:
git diff origin/asf-site
There are lots of spurious changes, such as timestamps and CSS styles in
tables. To see a list of files that have been added, deleted, renamed, changed
type, or are otherwise interesting, use the following command:
git diff --name-status --diff-filter=ADCRTXUB origin/asf-site
To see only files that had 100 or more lines changed:
git diff --stat origin/asf-site | grep -E '[1-9][0-9]{2,}'
When you are satisfied, publish your changes to origin/asf-site using this
command:
git push origin asf-site-f1d453599a90b8a1075082e2dc7b7676416f:asf-site
Changes take a couple of minutes to be propagated. You can then remove your
asf-site-f1d453599a90b8a1075082e2dc7b7676416f branch:
git checkout asf-site && git branch -d
asf-site-f1d453599a90b8a1075082e2dc7b7676416f
If failed, see https://builds.apache.org/job/hbase_generate_website/173/console
Successful: HBase Generate Website
Build status: Successful
If successful, the website and docs have been generated. If failed, skip to the
bottom of this email.
Use the following commands to download the patch and apply it to a clean branch
based on origin/asf-site. If you prefer to keep the hbase-site repo around
permanently, you can skip the clone step.
git clone https://git-wip-us.apache.org/repos/asf/hbase-site.git
cd hbase-site
wget -O-
https://builds.apache.org/job/hbase_generate_website/172/artifact/website.patch.zip
| funzip > f1d453599a90b8a1075082e2dc7b7676416f.patch
git fetch
git checkout -b asf-site-f1d453599a90b8a1075082e2dc7b7676416f
origin/asf-site
git am f1d453599a90b8a1075082e2dc7b7676416f.patch
At this point, you can preview the changes by opening index.html or any of the
other HTML pages in your local
asf-site-f1d453599a90b8a1075082e2dc7b7676416f branch, and you can review
the differences by running:
git diff origin/asf-site
There are lots of spurious changes, such as timestamps and CSS styles in
tables. To see a list of files that have been added, deleted, renamed, changed
type, or are otherwise interesting, use the following command:
git diff --name-status --diff-filter=ADCRTXUB origin/asf-site
To see only files that had 100 or more lines changed:
git diff --stat origin/asf-site | grep -E '[1-9][0-9]{2,}'
When you are satisfied, publish your changes to origin/asf-site using this
command:
git push origin asf-site-f1d453599a90b8a1075082e2dc7b7676416f:asf-site
Changes take a couple of minutes to be propagated. You can then remove your
asf-site-f1d453599a90b8a1075082e2dc7b7676416f branch:
git checkout asf-site && git branch -d
asf-site-f1d453599a90b8a1075082e2dc7b7676416f
If failed, see https://builds.apache.org/job/hbase_generate_website/172/console
[jira] [Created] (HBASE-15499) Add multiple data type support for increment
He Liangliang created HBASE-15499: - Summary: Add multiple data type support for increment Key: HBASE-15499 URL: https://issues.apache.org/jira/browse/HBASE-15499 Project: HBase Issue Type: New Feature Components: API Reporter: He Liangliang Currently the increment assumes long with byte-wise serialization. It's useful to support flexible data type/serializations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15498) Add conditions in MultiRowMutationService
He Liangliang created HBASE-15498:
-
Summary: Add conditions in MultiRowMutationService
Key: HBASE-15498
URL: https://issues.apache.org/jira/browse/HBASE-15498
Project: HBase
Issue Type: New Feature
Components: Coprocessors
Reporter: He Liangliang
Many applications especially online services need conditional batch operations
to achieve atomicity. We can add conditions in MultiRowMutationService like:
message MutateRowsRequest {
repeated MutationProto mutation_request = 1;
optional uint64 nonce_group = 2;
optional uint64 nonce = 3;
repeated Condition condition = 4;
}
message MutateRowsResponse {
repeated int32 unmet_conditions = 1;
}
service MultiRowMutationService {
rpc MutateRows(MutateRowsRequest)
returns(MutateRowsResponse);
}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-15497) Incorrect javadoc for atomicity guarantee of Increment and Append
Jianwei Cui created HBASE-15497:
---
Summary: Incorrect javadoc for atomicity guarantee of Increment
and Append
Key: HBASE-15497
URL: https://issues.apache.org/jira/browse/HBASE-15497
Project: HBase
Issue Type: Bug
Components: documentation
Affects Versions: 2.0.0
Reporter: Jianwei Cui
Priority: Minor
At the front of {{Increment.java}} file, there is comment about read atomicity:
{code}
* This operation does not appear atomic to readers. Increments are done
* under a single row lock, so write operations to a row are synchronized, but
* readers do not take row locks so get and scan operations can see this
* operation partially completed.
{code}
It seems this comment is not true after MVCC integrated
[HBASE-4583|https://issues.apache.org/jira/browse/HBASE-4583]. Currently, the
readers can be guaranteed to read the whole result of Increment if I am not
wrong. Similar comments also exist in {{Append.java}}, {{Table#append(...)}}
and {{Table#increment(...)}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-11684) HBase replicationSource should support multithread to ship the log entry
[ https://issues.apache.org/jira/browse/HBASE-11684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi resolved HBASE-11684. --- Resolution: Duplicate Assignee: (was: Tianying Chang) Duplicate of HBASE-12988 > HBase replicationSource should support multithread to ship the log entry > > > Key: HBASE-11684 > URL: https://issues.apache.org/jira/browse/HBASE-11684 > Project: HBase > Issue Type: Improvement > Components: Performance, Replication >Reporter: Tianying Chang > > We found the replication rate cannot keep up with the write rate when the > master cluster is write heavy. We got huge log queue build up due to that. > But when we do a rolling restart of master cluster, we found that the > appliedOpsRate doubled due to the extra thread created to help recover the > log of the restarted RS. ReplicateLogEntries is a synchronous blocking call, > it becomes the bottleneck when is only runs with one thread. I think we > should support multi-thread for the replication source to ship the data. I > don't see any consistency problem. Any other concern here? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15496) Throw RowTooBigException only for user scan/get
Guanghao Zhang created HBASE-15496: -- Summary: Throw RowTooBigException only for user scan/get Key: HBASE-15496 URL: https://issues.apache.org/jira/browse/HBASE-15496 Project: HBase Issue Type: Improvement Components: Scanners Reporter: Guanghao Zhang Priority: Minor Fix For: 2.0.0 When config hbase.table.max.rowsize, RowTooBigException may be thrown by StoreScanner. But region flush/compact should catch it or throw it only for user scan. org.apache.hadoop.hbase.regionserver.RowTooBigException: Max row size allowed: 10485760, but row is bigger than that at org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:355) at org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:276) at org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:238) at org.apache.hadoop.hbase.regionserver.compactions.Compactor.createScanner(Compactor.java:403) at org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:95) at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:131) at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1211) at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1952) at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1774) or org.apache.hadoop.hbase.regionserver.RowTooBigException: Max row size allowed: 10485760, but the row is bigger than that. at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:576) at org.apache.hadoop.hbase.regionserver.StoreFlusher.performFlush(StoreFlusher.java:132) at org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:75) at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:880) at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:2155) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2454) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2193) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2162) at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2053) at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1979) at org.apache.hadoop.hbase.regionserver.TestRowTooBig.testScannersSeekOnFewLargeCells(TestRowTooBig.java:101) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Successful: hbase.apache.org HTML Checker
Successful If successful, the HTML and link-checking report for http://hbase.apache.org is available at https://builds.apache.org/job/HBase%20Website%20Link%20Ckecker/31/artifact/link_report/index.html. If failed, see https://builds.apache.org/job/HBase%20Website%20Link%20Ckecker/31/console.
Re: Branch for 1.3
For Spark connector, we should start a separate discussion thread about backporting to branch-1. Zhan has a bug fix coming this week which deals with how negative numbers are handled in comparison. FYI On Mon, Mar 21, 2016 at 1:35 AM, Mikhail Antonov wrote: > Hi folks, > > bringing this topic up again. I'm planning to start spinning 1.3 builds and > see if/where they break in a week or two, and (depending on how it does) > start preparing RCs in a month or maybe two. So, let's see where we are. > > Big items first. There were long debates around three big items - MOBs, > Spark connector and RS groups, whether we should have them or not. > > - MOBs > I believe we decided that they aren't going to go in branch-1, and hence > not in branch-1.3 for sure. We might get back to that debate and > re-consider MOBs for branch-1 if 2.0 is delayed, but almost certainly they > won't make it in 1.3 timeframe anyway as I feel. > > - Spark connector - HBASE-14160 it looks like it has 3 subtasks open, one > of which is big one (HBASE-14375) - define public API for Spark > integration. From the Jira looks like active work is happening on other > subtasks, but not on this one. So how's public API going? How stable it is? > Who would want to have Spark in 1.3 and willing to help with this? OTOH - > who has objections about back-porting it? Has anyone been using it in some > real environment? > > - RS groups - there was recently a thread about them, I'd like to bring it > up again and get to some conclusion. > > Other features which we had in flight a month ago - > > - HBASE-15181 - date based tiered compactions has landed > - HBASE-11290 - unlock RegionStates. I'm afraid the codebase has moved > forward quite a bit since the benchmark was run on this change :( - Francis > - are you using it now? If we could have some benchmarks on the latest > rebase that I think would be great. > > > - HBASE-13557 - special handling for system tables WALs - should we still > keep it targeted for 1.3? > - HBASE-13017 - keep table state in meta, this one doesn't look like it's > going to make it in > > As a new item on my list - I'm looking forward to see more of HBASE-15492 > (memory optimizations) subtasks to go in branch-1. > > Thanks! > Mikhail > > On Sun, Feb 28, 2016 at 10:53 AM, Andrew Purtell > > wrote: > > > I'm starting a push at work to get us up on 1.2. Assuming that happens > > later this year I think that will be the end of my close attention to > 0.98. > > > > > On Feb 26, 2016, at 1:54 PM, Nick Dimiduk wrote: > > > > > > On Fri, Feb 26, 2016 at 12:23 PM, Andrew Purtell > > > wrote: > > > > > >> In the meantime those of us running HBase in production would benefit > > from > > >> fairly frequent minor releases. > > > > > > +1. Having to look back to 0.98 to get some new feature is problematic. > > > > > >> On Fri, Feb 26, 2016 at 9:50 AM, Elliott Clark > > wrote: > > >> > > >>> I disagree. We have agreed that 2.0 will have a new assignement > > manager. > > >>> There's a lot of work that has been done on getting that in, so far > > there > > >>> are no benefits to the end user from all that work. We should stick > > with > > >>> the plan and release 2.0 when it's ready. > > >>> > > >>> On Fri, Feb 26, 2016 at 9:39 AM, Stephen Jiang < > > syuanjiang...@gmail.com> > > >>> wrote: > > >>> > > Thanks for Mikhail for taking the 1.3 RM role. Looks like we have a > > >> lot > > >>> of > > new things in 1.3 release. > > > > Based on the experience of 1.1 and 1.2 release, it takes a lot of > > >> efforts > > to get a stable minor release out. From this, I have my own 2-cents > > on > > >>> 1.4 > > release. The plan is to have 2.0 release during summer time of this > > >> year > > (yeah, *this year). * Given the limited time and resource, after > 1.3 > > release, instead of spending effort on 1.4 release, the community > > >> should > > focus on stabilizing master (or branch-2, not exist as of now) > branch > > >> and > > make 2.0 release a priority. 2.0 release would bring more values to > > customer & move towards maturity of HBASE product. > > > > Thanks > > Stephen > > > > On Fri, Feb 26, 2016 at 1:49 AM, Mikhail Antonov < > > olorinb...@gmail.com > > >>> > > wrote: > > > > > Created an umbrella jira for 1.3 release - HBASE-15341 > > > > > > So it looks like we may have 1.4 release before 2.0 is out? I tried > > >> to > > add > > > 1.4 version in jira so we can keep it in branch-1 poms but I > > >> couldn't - > > > looks like I don't have permissions? > > > > > > -Mikhail > > > > > > On Thu, Feb 25, 2016 at 3:52 PM, Andrew Purtell < > apurt...@apache.org > > >>> > > > wrote: > > > > > >> The guy we had looking at streaming replication moved on and > > >> there's > > >>> no > > >> immediate plans to take on the work, FWIW > > >> > > >> On Thu, Feb 25, 2016 at 3:3
[jira] [Created] (HBASE-15495) Connection leak in FanOutOneBlockAsyncDFSOutputHelper
Duo Zhang created HBASE-15495: - Summary: Connection leak in FanOutOneBlockAsyncDFSOutputHelper Key: HBASE-15495 URL: https://issues.apache.org/jira/browse/HBASE-15495 Project: HBase Issue Type: Sub-task Reporter: Duo Zhang Assignee: Duo Zhang Happens when connecting to datanode failed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Branch for 1.3
Hi folks, bringing this topic up again. I'm planning to start spinning 1.3 builds and see if/where they break in a week or two, and (depending on how it does) start preparing RCs in a month or maybe two. So, let's see where we are. Big items first. There were long debates around three big items - MOBs, Spark connector and RS groups, whether we should have them or not. - MOBs I believe we decided that they aren't going to go in branch-1, and hence not in branch-1.3 for sure. We might get back to that debate and re-consider MOBs for branch-1 if 2.0 is delayed, but almost certainly they won't make it in 1.3 timeframe anyway as I feel. - Spark connector - HBASE-14160 it looks like it has 3 subtasks open, one of which is big one (HBASE-14375) - define public API for Spark integration. From the Jira looks like active work is happening on other subtasks, but not on this one. So how's public API going? How stable it is? Who would want to have Spark in 1.3 and willing to help with this? OTOH - who has objections about back-porting it? Has anyone been using it in some real environment? - RS groups - there was recently a thread about them, I'd like to bring it up again and get to some conclusion. Other features which we had in flight a month ago - - HBASE-15181 - date based tiered compactions has landed - HBASE-11290 - unlock RegionStates. I'm afraid the codebase has moved forward quite a bit since the benchmark was run on this change :( - Francis - are you using it now? If we could have some benchmarks on the latest rebase that I think would be great. - HBASE-13557 - special handling for system tables WALs - should we still keep it targeted for 1.3? - HBASE-13017 - keep table state in meta, this one doesn't look like it's going to make it in As a new item on my list - I'm looking forward to see more of HBASE-15492 (memory optimizations) subtasks to go in branch-1. Thanks! Mikhail On Sun, Feb 28, 2016 at 10:53 AM, Andrew Purtell wrote: > I'm starting a push at work to get us up on 1.2. Assuming that happens > later this year I think that will be the end of my close attention to 0.98. > > > On Feb 26, 2016, at 1:54 PM, Nick Dimiduk wrote: > > > > On Fri, Feb 26, 2016 at 12:23 PM, Andrew Purtell > > wrote: > > > >> In the meantime those of us running HBase in production would benefit > from > >> fairly frequent minor releases. > > > > +1. Having to look back to 0.98 to get some new feature is problematic. > > > >> On Fri, Feb 26, 2016 at 9:50 AM, Elliott Clark > wrote: > >> > >>> I disagree. We have agreed that 2.0 will have a new assignement > manager. > >>> There's a lot of work that has been done on getting that in, so far > there > >>> are no benefits to the end user from all that work. We should stick > with > >>> the plan and release 2.0 when it's ready. > >>> > >>> On Fri, Feb 26, 2016 at 9:39 AM, Stephen Jiang < > syuanjiang...@gmail.com> > >>> wrote: > >>> > Thanks for Mikhail for taking the 1.3 RM role. Looks like we have a > >> lot > >>> of > new things in 1.3 release. > > Based on the experience of 1.1 and 1.2 release, it takes a lot of > >> efforts > to get a stable minor release out. From this, I have my own 2-cents > on > >>> 1.4 > release. The plan is to have 2.0 release during summer time of this > >> year > (yeah, *this year). * Given the limited time and resource, after 1.3 > release, instead of spending effort on 1.4 release, the community > >> should > focus on stabilizing master (or branch-2, not exist as of now) branch > >> and > make 2.0 release a priority. 2.0 release would bring more values to > customer & move towards maturity of HBASE product. > > Thanks > Stephen > > On Fri, Feb 26, 2016 at 1:49 AM, Mikhail Antonov < > olorinb...@gmail.com > >>> > wrote: > > > Created an umbrella jira for 1.3 release - HBASE-15341 > > > > So it looks like we may have 1.4 release before 2.0 is out? I tried > >> to > add > > 1.4 version in jira so we can keep it in branch-1 poms but I > >> couldn't - > > looks like I don't have permissions? > > > > -Mikhail > > > > On Thu, Feb 25, 2016 at 3:52 PM, Andrew Purtell >>> > > wrote: > > > >> The guy we had looking at streaming replication moved on and > >> there's > >>> no > >> immediate plans to take on the work, FWIW > >> > >> On Thu, Feb 25, 2016 at 3:33 PM, Matteo Bertozzi < > > theo.berto...@gmail.com> > >> wrote: > >> > >>> I was shooting for summer for hbase 2.0, the main problem is that > there > >> is > >>> still no code for the new AM or for fs changes, which are the two > that > >> may > >>> impact compatibility (working slowly on that). Streaming > >>> replication > > and > >>> others seems compatible enough but no code there too. > >>> > >>> Matteo > >>> > >>> > >>> On Thu, Feb 25, 2016 at 3:29 PM, Mi
