Fixed: HBase Generate Website
Build status: Fixed If successful, the website and docs have been generated. If failed, skip to the bottom of this email. Use the following commands to download the patch and apply it to a clean branch based on origin/asf-site. If you prefer to keep the hbase-site repo around permanently, you can skip the clone step. git clone https://git-wip-us.apache.org/repos/asf/hbase-site.git cd hbase-site wget -O- https://builds.apache.org/job/hbase_generate_website/92/artifact/website.patch.zip | funzip > 893a54c3a40e08b6f50d753fca01688f491893d0.patch git fetch git checkout -b asf-site-893a54c3a40e08b6f50d753fca01688f491893d0 origin/asf-site git am 893a54c3a40e08b6f50d753fca01688f491893d0.patch At this point, you can preview the changes by opening index.html or any of the other HTML pages in your local asf-site-893a54c3a40e08b6f50d753fca01688f491893d0 branch, and you can review the differences by running: git diff origin/asf-site There are lots of spurious changes, such as timestamps and CSS styles in tables. To see a list of files that have been added, deleted, renamed, changed type, or are otherwise interesting, use the following command: git diff --name-status --diff-filter=ADCRTXUB origin/asf-site To see only files that had 10 or more lines changed: git diff --stat origin/asf-site | grep -Ev "\|\s+\ [1-9]\ [\+-]+$" When you are satisfied, publish your changes to origin/asf-site using this command: git push origin asf-site-893a54c3a40e08b6f50d753fca01688f491893d0:asf-site Changes take a couple of minutes to be propagated. You can then remove your asf-site-893a54c3a40e08b6f50d753fca01688f491893d0 branch: git checkout asf-site && git branch -d asf-site-893a54c3a40e08b6f50d753fca01688f491893d0 If failed, see https://builds.apache.org/job/hbase_generate_website/92/console
[jira] [Resolved] (HBASE-14107) Procedure V2 - Administrative Task: Provide an API to List all procedures
[ https://issues.apache.org/jira/browse/HBASE-14107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang resolved HBASE-14107. Resolution: Fixed Fix Version/s: 1.1.3 1.2.0 > Procedure V2 - Administrative Task: Provide an API to List all procedures > -- > > Key: HBASE-14107 > URL: https://issues.apache.org/jira/browse/HBASE-14107 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0, 1.3.0 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang > Labels: admin > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3 > > Attachments: HBASE-14107.v1-branch-1.1.patch, > HBASE-14107.v1-branch-1.2.patch, HBASE-14107.v1-branch-1.patch, > HBASE-14107.v1-master.patch, HBASE-14107.v2-master.patch > > > With Procedure V2 in production since HBASE 1.1 release, there is a need to > list all procedures (running, queued, recently completed) from HBASE shell > (or Web UI). > This JIRA is to track the work to add a API to list all procedures. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14432) Procedure V2 - enforce ACL on procedure admin tasks
[ https://issues.apache.org/jira/browse/HBASE-14432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang resolved HBASE-14432. Resolution: Fixed Fix Version/s: 1.1.3 1.2.0 > Procedure V2 - enforce ACL on procedure admin tasks > --- > > Key: HBASE-14432 > URL: https://issues.apache.org/jira/browse/HBASE-14432 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0, 1.3.0 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang > Labels: security > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3 > > Attachments: HBASE-14432.v1-branch-1.1.patch, > HBASE-14432.v1-branch-1.2.patch, HBASE-14432.v1-branch-1.patch, > HBASE-14432.v1-master.patch > > > In the Procedure class, the owner field is never set. We need to set it so > that we can enforce ACLs on admin tasks such as whether a user has privilege > to abort a procedure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14108) Procedure V2 - Administrative Task: provide an API to abort a procedure
[ https://issues.apache.org/jira/browse/HBASE-14108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang resolved HBASE-14108. Resolution: Fixed Fix Version/s: 1.1.3 1.2.0 > Procedure V2 - Administrative Task: provide an API to abort a procedure > --- > > Key: HBASE-14108 > URL: https://issues.apache.org/jira/browse/HBASE-14108 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0, 1.3.0 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3 > > Attachments: HBASE-14108.v1-branch-1.1.patch, > HBASE-14108.v1-branch-1.2.patch, HBASE-14108.v1-branch-1.patch, > HBASE-14108.v1-master.patch, HBASE-14108.v2-master.patch > > > With Procedure-V2 in production since HBASE 1.1 release, there is a need to > abort a procedure (eg. for a long-running procedure that stucks somewhere and > blocks others). The command could either from shell or Web UI. > This task tracks the work to provide an API to abort a procedure (either > rollback or simply quit). This API could be used either from shell or Web UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14487) Procedure V2 - shell command to list all procedures
[ https://issues.apache.org/jira/browse/HBASE-14487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang resolved HBASE-14487. Resolution: Fixed > Procedure V2 - shell command to list all procedures > --- > > Key: HBASE-14487 > URL: https://issues.apache.org/jira/browse/HBASE-14487 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, shell >Affects Versions: 2.0.0, 1.3.0 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3 > > Attachments: HBASE-14487.patch > > > Use the API provided by HBASE-14107, add a shell command to list all > procedures. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14488) Procedure V2 - shell command to abort a procedure
[ https://issues.apache.org/jira/browse/HBASE-14488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang resolved HBASE-14488. Resolution: Fixed Fix Version/s: 1.1.3 1.2.0 > Procedure V2 - shell command to abort a procedure > - > > Key: HBASE-14488 > URL: https://issues.apache.org/jira/browse/HBASE-14488 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0, 1.3.0 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang > Labels: shell > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3 > > Attachments: abortProcedureShell.v1-master.patch > > > Use the API provided by HBASE-14108, add a shell command to abort a specific > procedure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [VOTE] HBase 1.2.0 RC0
Please also consider including HBASE-15073 in the next RC. This JIRA gives finer grained control over region normalization. Since 1.2.0 would be the first release with region normalization feature, it would be nice to get user facing aspect straightened out. Thanks On Tue, Jan 5, 2016 at 1:47 PM, Elliott Clark wrote: > https://issues.apache.org/jira/browse/HBASE-14962 > > Is still rearing its ugly head on me. TestSplitWalDataLoss fails on me > every 1/4 times. > > On Tue, Jan 5, 2016 at 12:34 PM, Sean Busbey wrote: > > > Anyone have objections to including the proc-v2 admin tasks Stephen > > mentions? > > > > I should have time tonight to spin an updated RC. These changes seem > > relatively isolated. Anyone currently testing that would oppose me > keeping > > the same closing date for the updated candidate? > > > > On Tue, Jan 5, 2016 at 2:09 PM, Stephen Jiang > > wrote: > > > > > The administrative tasks for proc-v2 (5 JIRAs: HBASE-14107, > HBASE-14108, > > > HBASE-14487, HBASE-14488, and HBASE-14432) would be useful for > debugging > > > or unblocking hanging procedures in master DDL. The change is local > and > > > low risk. Discussing with Sean and Matteo privately, they agreed to > > > include them in 1.2. I will commit to branch-1.2 today so that the > next > > > 1.2RC would have the changes. > > > > > > thanks > > > Stephen > > > > > > > > > > > > On Mon, Jan 4, 2016 at 3:16 PM, Ted Yu wrote: > > > > > >> +1 > > >> > > >> Ran unit test suite > > >> Exercised basic shell commands > > >> > > >> [INFO] BUILD SUCCESS > > >> [INFO] > > >> > > > >> [INFO] Total time: 01:44 h > > >> [INFO] Finished at: 2016-01-03T09:12:35-08:00 > > >> > > >> On Sun, Jan 3, 2016 at 4:29 AM, Sean Busbey > wrote: > > >> > > >> > Happy New Year everyone! I'm pleased to announce the first release > > >> > candidate for HBase 1.2.0. > > >> > > > >> > Artifacts are available here: > > >> > > > >> > https://dist.apache.org/repos/dist/dev/hbase/hbase-1.2.0RC0/ > > >> > > > >> > As of this vote, the relevant md5 hashes are: > > >> > 41c499a325ae6449fe99bbe315d07e10 hbase-1.2.0-bin.tar.gz > > >> > 537874610f1b53d4bce9d8a925661246 hbase-1.2.0-src.tar.gz > > >> > > > >> > Maven artifacts are also available in the staging repository: > > >> > > > >> > > > https://repository.apache.org/content/repositories/orgapachehbase-1122/ > > >> > > > >> > artifacts are signed with my code signing key 0D80DB7C, available in > > >> > the project KEYS file: > > >> > > > >> > http://www.apache.org/dist/hbase/KEYS > > >> > > > >> > these artifacts correspond to commit hash > > >> > > > >> > 20c4368065165ad49bdfe8172316e42566a6d6a0 > > >> > > > >> > which signed tag 1.2.0RC0 currently points to > > >> > > > >> > > > >> > > > >> > > > https://git1-us-west.apache.org/repos/asf?p=hbase.git;a=tag;h=771e13960cf406185bb29447dffb6f2b5a56baec > > >> > > > >> > HBase 1.2.0 is the second minor release in the HBase 1.x line, > > >> continuing > > >> > on > > >> > the theme of bringing a stable, reliable database to the Hadoop and > > >> NoSQL > > >> > communities. This release includes roughly 250 resolved issues not > > >> covered > > >> > by previous 1.x releases. > > >> > > > >> > Notable new features include: > > >> > - JDK8 is now supported > > >> > - Hadoop 2.6.1+ and Hadoop 2.7.1+ are now supported > > >> > - per column-family time ranges for scan (HBASE-14355) > > >> > - daemons respond to SIGHUP to reload configs (HBASE-14529) > > >> > - region location methods added to thrift2 proxy (HBASE-13698) > > >> > - table-level sync that sends deltas (HBASE-13639) > > >> > - client side metrics via JMX (HBASE-12911) > > >> > > > >> > The full list of issues can be found at: > > >> > > > >> > > > >> > > > >> > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12332062 > > >> > > > >> > Please take a few minutes to verify the release[1] and vote on > > releasing > > >> > it: > > >> > > > >> > [ ] +1 Release this package as Apache HBase 1.2.0 > > >> > [ ] +0 no opinion > > >> > [ ] -1 Do not release this package because... > > >> > > > >> > Vote will be subject to Majority Approval[2] and will close at > 8:00PM > > >> > UTC on Monday, Jan 11th, 2016[3]. > > >> > > > >> > [1]: http://www.apache.org/info/verification.html > > >> > [2]: > https://www.apache.org/foundation/glossary.html#MajorityApproval > > >> > [3]: to find this in your local timezone see: > > >> > http://s.apache.org/hbase-1.2.0-rc0-close > > >> > > > >> > > > > > > > > > > > > -- > > Sean > > >
[jira] [Created] (HBASE-15074) Zombie maker to test zombie detector
stack created HBASE-15074: - Summary: Zombie maker to test zombie detector Key: HBASE-15074 URL: https://issues.apache.org/jira/browse/HBASE-15074 Project: HBase Issue Type: Sub-task Reporter: stack Our Sean thinks our zombie detector is not finding zombies (and I think he is right). Here is a test that makes zombies. Lets see what happens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [VOTE] HBase 1.2.0 RC0
+1 to include patches which improve region normalization. Since it's new functionality for 1.2, should be low risk (no regressions). -Mikhail On Wed, Jan 6, 2016 at 8:37 AM, Ted Yu wrote: > Please also consider including HBASE-15073 in the next RC. > > This JIRA gives finer grained control over region normalization. > Since 1.2.0 would be the first release with region normalization feature, > it would be nice to get user facing aspect straightened out. > > Thanks > > On Tue, Jan 5, 2016 at 1:47 PM, Elliott Clark wrote: > > > https://issues.apache.org/jira/browse/HBASE-14962 > > > > Is still rearing its ugly head on me. TestSplitWalDataLoss fails on me > > every 1/4 times. > > > > On Tue, Jan 5, 2016 at 12:34 PM, Sean Busbey > wrote: > > > > > Anyone have objections to including the proc-v2 admin tasks Stephen > > > mentions? > > > > > > I should have time tonight to spin an updated RC. These changes seem > > > relatively isolated. Anyone currently testing that would oppose me > > keeping > > > the same closing date for the updated candidate? > > > > > > On Tue, Jan 5, 2016 at 2:09 PM, Stephen Jiang > > > > wrote: > > > > > > > The administrative tasks for proc-v2 (5 JIRAs: HBASE-14107, > > HBASE-14108, > > > > HBASE-14487, HBASE-14488, and HBASE-14432) would be useful for > > debugging > > > > or unblocking hanging procedures in master DDL. The change is local > > and > > > > low risk. Discussing with Sean and Matteo privately, they agreed to > > > > include them in 1.2. I will commit to branch-1.2 today so that the > > next > > > > 1.2RC would have the changes. > > > > > > > > thanks > > > > Stephen > > > > > > > > > > > > > > > > On Mon, Jan 4, 2016 at 3:16 PM, Ted Yu wrote: > > > > > > > >> +1 > > > >> > > > >> Ran unit test suite > > > >> Exercised basic shell commands > > > >> > > > >> [INFO] BUILD SUCCESS > > > >> [INFO] > > > >> > > > > > >> [INFO] Total time: 01:44 h > > > >> [INFO] Finished at: 2016-01-03T09:12:35-08:00 > > > >> > > > >> On Sun, Jan 3, 2016 at 4:29 AM, Sean Busbey > > wrote: > > > >> > > > >> > Happy New Year everyone! I'm pleased to announce the first release > > > >> > candidate for HBase 1.2.0. > > > >> > > > > >> > Artifacts are available here: > > > >> > > > > >> > https://dist.apache.org/repos/dist/dev/hbase/hbase-1.2.0RC0/ > > > >> > > > > >> > As of this vote, the relevant md5 hashes are: > > > >> > 41c499a325ae6449fe99bbe315d07e10 hbase-1.2.0-bin.tar.gz > > > >> > 537874610f1b53d4bce9d8a925661246 hbase-1.2.0-src.tar.gz > > > >> > > > > >> > Maven artifacts are also available in the staging repository: > > > >> > > > > >> > > > > > https://repository.apache.org/content/repositories/orgapachehbase-1122/ > > > >> > > > > >> > artifacts are signed with my code signing key 0D80DB7C, available > in > > > >> > the project KEYS file: > > > >> > > > > >> > http://www.apache.org/dist/hbase/KEYS > > > >> > > > > >> > these artifacts correspond to commit hash > > > >> > > > > >> > 20c4368065165ad49bdfe8172316e42566a6d6a0 > > > >> > > > > >> > which signed tag 1.2.0RC0 currently points to > > > >> > > > > >> > > > > >> > > > > >> > > > > > > https://git1-us-west.apache.org/repos/asf?p=hbase.git;a=tag;h=771e13960cf406185bb29447dffb6f2b5a56baec > > > >> > > > > >> > HBase 1.2.0 is the second minor release in the HBase 1.x line, > > > >> continuing > > > >> > on > > > >> > the theme of bringing a stable, reliable database to the Hadoop > and > > > >> NoSQL > > > >> > communities. This release includes roughly 250 resolved issues not > > > >> covered > > > >> > by previous 1.x releases. > > > >> > > > > >> > Notable new features include: > > > >> > - JDK8 is now supported > > > >> > - Hadoop 2.6.1+ and Hadoop 2.7.1+ are now supported > > > >> > - per column-family time ranges for scan (HBASE-14355) > > > >> > - daemons respond to SIGHUP to reload configs (HBASE-14529) > > > >> > - region location methods added to thrift2 proxy (HBASE-13698) > > > >> > - table-level sync that sends deltas (HBASE-13639) > > > >> > - client side metrics via JMX (HBASE-12911) > > > >> > > > > >> > The full list of issues can be found at: > > > >> > > > > >> > > > > >> > > > > >> > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12332062 > > > >> > > > > >> > Please take a few minutes to verify the release[1] and vote on > > > releasing > > > >> > it: > > > >> > > > > >> > [ ] +1 Release this package as Apache HBase 1.2.0 > > > >> > [ ] +0 no opinion > > > >> > [ ] -1 Do not release this package because... > > > >> > > > > >> > Vote will be subject to Majority Approval[2] and will close at > > 8:00PM > > > >> > UTC on Monday, Jan 11th, 2016[3]. > > > >> > > > > >> > [1]: http://www.apache.org/info/verification.html > > > >> > [2]: > > https://www.apache.org/foundation/glossary.html#MajorityApproval > > > >> > [3]: to find this in your local timezone see: > > > >> > http://s.apache
Re: [VOTE] HBase 1.2.0 RC0
Folks, please comment on the relevant jiras and not on the VOTE thread. On Wed, Jan 6, 2016 at 11:49 AM, Mikhail Antonov wrote: > +1 to include patches which improve region normalization. Since it's new > functionality for 1.2, should be low risk (no regressions). > > -Mikhail > > On Wed, Jan 6, 2016 at 8:37 AM, Ted Yu wrote: > > > Please also consider including HBASE-15073 in the next RC. > > > > This JIRA gives finer grained control over region normalization. > > Since 1.2.0 would be the first release with region normalization feature, > > it would be nice to get user facing aspect straightened out. > > > > Thanks > > > > On Tue, Jan 5, 2016 at 1:47 PM, Elliott Clark wrote: > > > > > https://issues.apache.org/jira/browse/HBASE-14962 > > > > > > Is still rearing its ugly head on me. TestSplitWalDataLoss fails on me > > > every 1/4 times. > > > > > > On Tue, Jan 5, 2016 at 12:34 PM, Sean Busbey > > wrote: > > > > > > > Anyone have objections to including the proc-v2 admin tasks Stephen > > > > mentions? > > > > > > > > I should have time tonight to spin an updated RC. These changes seem > > > > relatively isolated. Anyone currently testing that would oppose me > > > keeping > > > > the same closing date for the updated candidate? > > > > > > > > On Tue, Jan 5, 2016 at 2:09 PM, Stephen Jiang < > syuanjiang...@gmail.com > > > > > > > wrote: > > > > > > > > > The administrative tasks for proc-v2 (5 JIRAs: HBASE-14107, > > > HBASE-14108, > > > > > HBASE-14487, HBASE-14488, and HBASE-14432) would be useful for > > > debugging > > > > > or unblocking hanging procedures in master DDL. The change is > local > > > and > > > > > low risk. Discussing with Sean and Matteo privately, they agreed > to > > > > > include them in 1.2. I will commit to branch-1.2 today so that the > > > next > > > > > 1.2RC would have the changes. > > > > > > > > > > thanks > > > > > Stephen > > > > > > > > > > > > > > > > > > > > On Mon, Jan 4, 2016 at 3:16 PM, Ted Yu > wrote: > > > > > > > > > >> +1 > > > > >> > > > > >> Ran unit test suite > > > > >> Exercised basic shell commands > > > > >> > > > > >> [INFO] BUILD SUCCESS > > > > >> [INFO] > > > > >> > > > > > > > > >> [INFO] Total time: 01:44 h > > > > >> [INFO] Finished at: 2016-01-03T09:12:35-08:00 > > > > >> > > > > >> On Sun, Jan 3, 2016 at 4:29 AM, Sean Busbey > > > wrote: > > > > >> > > > > >> > Happy New Year everyone! I'm pleased to announce the first > release > > > > >> > candidate for HBase 1.2.0. > > > > >> > > > > > >> > Artifacts are available here: > > > > >> > > > > > >> > https://dist.apache.org/repos/dist/dev/hbase/hbase-1.2.0RC0/ > > > > >> > > > > > >> > As of this vote, the relevant md5 hashes are: > > > > >> > 41c499a325ae6449fe99bbe315d07e10 hbase-1.2.0-bin.tar.gz > > > > >> > 537874610f1b53d4bce9d8a925661246 hbase-1.2.0-src.tar.gz > > > > >> > > > > > >> > Maven artifacts are also available in the staging repository: > > > > >> > > > > > >> > > > > > > > https://repository.apache.org/content/repositories/orgapachehbase-1122/ > > > > >> > > > > > >> > artifacts are signed with my code signing key 0D80DB7C, > available > > in > > > > >> > the project KEYS file: > > > > >> > > > > > >> > http://www.apache.org/dist/hbase/KEYS > > > > >> > > > > > >> > these artifacts correspond to commit hash > > > > >> > > > > > >> > 20c4368065165ad49bdfe8172316e42566a6d6a0 > > > > >> > > > > > >> > which signed tag 1.2.0RC0 currently points to > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > > > > > > https://git1-us-west.apache.org/repos/asf?p=hbase.git;a=tag;h=771e13960cf406185bb29447dffb6f2b5a56baec > > > > >> > > > > > >> > HBase 1.2.0 is the second minor release in the HBase 1.x line, > > > > >> continuing > > > > >> > on > > > > >> > the theme of bringing a stable, reliable database to the Hadoop > > and > > > > >> NoSQL > > > > >> > communities. This release includes roughly 250 resolved issues > not > > > > >> covered > > > > >> > by previous 1.x releases. > > > > >> > > > > > >> > Notable new features include: > > > > >> > - JDK8 is now supported > > > > >> > - Hadoop 2.6.1+ and Hadoop 2.7.1+ are now supported > > > > >> > - per column-family time ranges for scan (HBASE-14355) > > > > >> > - daemons respond to SIGHUP to reload configs (HBASE-14529) > > > > >> > - region location methods added to thrift2 proxy (HBASE-13698) > > > > >> > - table-level sync that sends deltas (HBASE-13639) > > > > >> > - client side metrics via JMX (HBASE-12911) > > > > >> > > > > > >> > The full list of issues can be found at: > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12332062 > > > > >> > > > > > >> > Please take a few minutes to verify the release[1] and vote on > > > > releasing > > > > >> > it: > > > > >> > > > > > >> > [ ] +1 Release this package as Apache HBase 1.2.0 > > > > >> > [
Re: [VOTE] HBase 1.2.0 RC0
You're right, sorry. On Wed, Jan 6, 2016 at 10:09 AM, Sean Busbey wrote: > Folks, please comment on the relevant jiras and not on the VOTE thread. > > On Wed, Jan 6, 2016 at 11:49 AM, Mikhail Antonov > wrote: > > > +1 to include patches which improve region normalization. Since it's new > > functionality for 1.2, should be low risk (no regressions). > > > > -Mikhail > > > > On Wed, Jan 6, 2016 at 8:37 AM, Ted Yu wrote: > > > > > Please also consider including HBASE-15073 in the next RC. > > > > > > This JIRA gives finer grained control over region normalization. > > > Since 1.2.0 would be the first release with region normalization > feature, > > > it would be nice to get user facing aspect straightened out. > > > > > > Thanks > > > > > > On Tue, Jan 5, 2016 at 1:47 PM, Elliott Clark > wrote: > > > > > > > https://issues.apache.org/jira/browse/HBASE-14962 > > > > > > > > Is still rearing its ugly head on me. TestSplitWalDataLoss fails on > me > > > > every 1/4 times. > > > > > > > > On Tue, Jan 5, 2016 at 12:34 PM, Sean Busbey > > > wrote: > > > > > > > > > Anyone have objections to including the proc-v2 admin tasks Stephen > > > > > mentions? > > > > > > > > > > I should have time tonight to spin an updated RC. These changes > seem > > > > > relatively isolated. Anyone currently testing that would oppose me > > > > keeping > > > > > the same closing date for the updated candidate? > > > > > > > > > > On Tue, Jan 5, 2016 at 2:09 PM, Stephen Jiang < > > syuanjiang...@gmail.com > > > > > > > > > wrote: > > > > > > > > > > > The administrative tasks for proc-v2 (5 JIRAs: HBASE-14107, > > > > HBASE-14108, > > > > > > HBASE-14487, HBASE-14488, and HBASE-14432) would be useful for > > > > debugging > > > > > > or unblocking hanging procedures in master DDL. The change is > > local > > > > and > > > > > > low risk. Discussing with Sean and Matteo privately, they agreed > > to > > > > > > include them in 1.2. I will commit to branch-1.2 today so that > the > > > > next > > > > > > 1.2RC would have the changes. > > > > > > > > > > > > thanks > > > > > > Stephen > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Jan 4, 2016 at 3:16 PM, Ted Yu > > wrote: > > > > > > > > > > > >> +1 > > > > > >> > > > > > >> Ran unit test suite > > > > > >> Exercised basic shell commands > > > > > >> > > > > > >> [INFO] BUILD SUCCESS > > > > > >> [INFO] > > > > > >> > > > > > > > > > > > >> [INFO] Total time: 01:44 h > > > > > >> [INFO] Finished at: 2016-01-03T09:12:35-08:00 > > > > > >> > > > > > >> On Sun, Jan 3, 2016 at 4:29 AM, Sean Busbey > > > > wrote: > > > > > >> > > > > > >> > Happy New Year everyone! I'm pleased to announce the first > > release > > > > > >> > candidate for HBase 1.2.0. > > > > > >> > > > > > > >> > Artifacts are available here: > > > > > >> > > > > > > >> > https://dist.apache.org/repos/dist/dev/hbase/hbase-1.2.0RC0/ > > > > > >> > > > > > > >> > As of this vote, the relevant md5 hashes are: > > > > > >> > 41c499a325ae6449fe99bbe315d07e10 hbase-1.2.0-bin.tar.gz > > > > > >> > 537874610f1b53d4bce9d8a925661246 hbase-1.2.0-src.tar.gz > > > > > >> > > > > > > >> > Maven artifacts are also available in the staging repository: > > > > > >> > > > > > > >> > > > > > > > > > > https://repository.apache.org/content/repositories/orgapachehbase-1122/ > > > > > >> > > > > > > >> > artifacts are signed with my code signing key 0D80DB7C, > > available > > > in > > > > > >> > the project KEYS file: > > > > > >> > > > > > > >> > http://www.apache.org/dist/hbase/KEYS > > > > > >> > > > > > > >> > these artifacts correspond to commit hash > > > > > >> > > > > > > >> > 20c4368065165ad49bdfe8172316e42566a6d6a0 > > > > > >> > > > > > > >> > which signed tag 1.2.0RC0 currently points to > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > https://git1-us-west.apache.org/repos/asf?p=hbase.git;a=tag;h=771e13960cf406185bb29447dffb6f2b5a56baec > > > > > >> > > > > > > >> > HBase 1.2.0 is the second minor release in the HBase 1.x line, > > > > > >> continuing > > > > > >> > on > > > > > >> > the theme of bringing a stable, reliable database to the > Hadoop > > > and > > > > > >> NoSQL > > > > > >> > communities. This release includes roughly 250 resolved issues > > not > > > > > >> covered > > > > > >> > by previous 1.x releases. > > > > > >> > > > > > > >> > Notable new features include: > > > > > >> > - JDK8 is now supported > > > > > >> > - Hadoop 2.6.1+ and Hadoop 2.7.1+ are now supported > > > > > >> > - per column-family time ranges for scan (HBASE-14355) > > > > > >> > - daemons respond to SIGHUP to reload configs (HBASE-14529) > > > > > >> > - region location methods added to thrift2 proxy (HBASE-13698) > > > > > >> > - table-level sync that sends deltas (HBASE-13639) > > > > > >> > - client side metrics via JMX (HBASE-12911) > > > > > >> > > > > > > >> > The full list of issues can be found at: > > >
Re: identifying source of region split request
Adding this tracing information to know who initiated split is in general useful thing. Right now though I'm not sure I see how that would help to make better normalization decisions? Region split failure implies underlying FS issue? Any examples/ideas? Kind of..collect statistics to blacklist some RSs with high failure rate and don't attempt to split regions hosted there in future? On Tue, Jan 5, 2016 at 2:55 PM, Ted Yu wrote: > Hi, > I recently worked on improving region normalization feature. > > If region split request triggered by the execution of > SplitNormalizationPlan fails, there is no way of knowing whether the failed > split originated from region normalization. > Such association would give RegionNormalizer information so that it can > make better normalization decisions in the subsequent invocations. > > One enhancement I can think of is to embed metadata in SplitRequest which > gets passed through RegionStateTransitionContext when > RegionServerServices#reportRegionStateTransition() is called. > This way, RegionStateListener can be notified with the metadata (id of the > requester). > > Comment is welcome. > -- Thanks, Michael Antonov
Re: identifying source of region split request
bq. collect statistics to blacklist some RSs with high failure rate The metadata would help pinpoint the regions which consistently fail split (merge) in the recent past. The failure could be due to corrupt HFile(s) or other reason. Having the statistics would also help normalizer avoid the following scenario: region R gets split into R1 and R2 size of R1 and R2 decreases due to expiration of data R1 and R2 get merged into R' more data comes into R', resulting in split, again Cheers On Wed, Jan 6, 2016 at 10:14 AM, Mikhail Antonov wrote: > Adding this tracing information to know who initiated split is in general > useful thing. Right now though I'm not sure I see how that would help to > make better normalization decisions? Region split failure implies > underlying FS issue? Any examples/ideas? > > Kind of..collect statistics to blacklist some RSs with high failure rate > and don't attempt to split regions hosted there in future? > > On Tue, Jan 5, 2016 at 2:55 PM, Ted Yu wrote: > > > Hi, > > I recently worked on improving region normalization feature. > > > > If region split request triggered by the execution of > > SplitNormalizationPlan fails, there is no way of knowing whether the > failed > > split originated from region normalization. > > Such association would give RegionNormalizer information so that it can > > make better normalization decisions in the subsequent invocations. > > > > One enhancement I can think of is to embed metadata in SplitRequest which > > gets passed through RegionStateTransitionContext when > > RegionServerServices#reportRegionStateTransition() is called. > > This way, RegionStateListener can be notified with the metadata (id of > the > > requester). > > > > Comment is welcome. > > > > > > -- > Thanks, > Michael Antonov >
Re: identifying source of region split request
Yeah, I see. Btw, in scenario you described, is splitting and subsequent merging a necessarily bad thing? If data expiration happens not that often, having more evenly sized regions in between major collections isn't worth it? On Wed, Jan 6, 2016 at 10:23 AM, Ted Yu wrote: > bq. collect statistics to blacklist some RSs with high failure rate > > The metadata would help pinpoint the regions which consistently fail split > (merge) in the recent past. The failure could be due to corrupt HFile(s) or > other reason. > Having the statistics would also help normalizer avoid the following > scenario: > > region R gets split into R1 and R2 > size of R1 and R2 decreases due to expiration of data > R1 and R2 get merged into R' > more data comes into R', resulting in split, again > > Cheers > > On Wed, Jan 6, 2016 at 10:14 AM, Mikhail Antonov > wrote: > > > Adding this tracing information to know who initiated split is in general > > useful thing. Right now though I'm not sure I see how that would help to > > make better normalization decisions? Region split failure implies > > underlying FS issue? Any examples/ideas? > > > > Kind of..collect statistics to blacklist some RSs with high failure rate > > and don't attempt to split regions hosted there in future? > > > > On Tue, Jan 5, 2016 at 2:55 PM, Ted Yu wrote: > > > > > Hi, > > > I recently worked on improving region normalization feature. > > > > > > If region split request triggered by the execution of > > > SplitNormalizationPlan fails, there is no way of knowing whether the > > failed > > > split originated from region normalization. > > > Such association would give RegionNormalizer information so that it can > > > make better normalization decisions in the subsequent invocations. > > > > > > One enhancement I can think of is to embed metadata in SplitRequest > which > > > gets passed through RegionStateTransitionContext when > > > RegionServerServices#reportRegionStateTransition() is called. > > > This way, RegionStateListener can be notified with the metadata (id of > > the > > > requester). > > > > > > Comment is welcome. > > > > > > > > > > > -- > > Thanks, > > Michael Antonov > > > -- Thanks, Michael Antonov
Re: identifying source of region split request
bq. is splitting and subsequent merging a necessarily bad thing? Please note that merging may happen before the size of R1 (R2) drops to zero. Meaning, there may be churn in normalization activity. I would say that the normalization in the scenario below is premature. Cheers On Wed, Jan 6, 2016 at 11:41 AM, Mikhail Antonov wrote: > Yeah, I see. Btw, in scenario you described, is splitting and subsequent > merging a necessarily bad thing? If data expiration happens not that often, > having more evenly sized regions in between major collections isn't worth > it? > > On Wed, Jan 6, 2016 at 10:23 AM, Ted Yu wrote: > > > bq. collect statistics to blacklist some RSs with high failure rate > > > > The metadata would help pinpoint the regions which consistently fail > split > > (merge) in the recent past. The failure could be due to corrupt HFile(s) > or > > other reason. > > Having the statistics would also help normalizer avoid the following > > scenario: > > > > region R gets split into R1 and R2 > > size of R1 and R2 decreases due to expiration of data > > R1 and R2 get merged into R' > > more data comes into R', resulting in split, again > > > > Cheers > > > > On Wed, Jan 6, 2016 at 10:14 AM, Mikhail Antonov > > wrote: > > > > > Adding this tracing information to know who initiated split is in > general > > > useful thing. Right now though I'm not sure I see how that would help > to > > > make better normalization decisions? Region split failure implies > > > underlying FS issue? Any examples/ideas? > > > > > > Kind of..collect statistics to blacklist some RSs with high failure > rate > > > and don't attempt to split regions hosted there in future? > > > > > > On Tue, Jan 5, 2016 at 2:55 PM, Ted Yu wrote: > > > > > > > Hi, > > > > I recently worked on improving region normalization feature. > > > > > > > > If region split request triggered by the execution of > > > > SplitNormalizationPlan fails, there is no way of knowing whether the > > > failed > > > > split originated from region normalization. > > > > Such association would give RegionNormalizer information so that it > can > > > > make better normalization decisions in the subsequent invocations. > > > > > > > > One enhancement I can think of is to embed metadata in SplitRequest > > which > > > > gets passed through RegionStateTransitionContext when > > > > RegionServerServices#reportRegionStateTransition() is called. > > > > This way, RegionStateListener can be notified with the metadata (id > of > > > the > > > > requester). > > > > > > > > Comment is welcome. > > > > > > > > > > > > > > > > -- > > > Thanks, > > > Michael Antonov > > > > > > > > > -- > Thanks, > Michael Antonov >
Re: Compact after manual flush broken in 1.0+?
Unable to find it. Feel free to open. JMS 2015-12-31 12:36 GMT-05:00 Jean-Marc Spaggiari : > I think there is one. I will search later today... > Le 2015-12-31 11:35 AM, "Ted Yu" a écrit : > >> A brief search turned up no hit. >> >> Please go ahead with new JIRA. >> >> On Thu, Dec 31, 2015 at 8:32 AM, wrote: >> >> > Cool :) >> > Is there a jira already? Otherwise I'll file one. >> > -- Lars >> > >> > From: Jean-Marc Spaggiari >> > To: dev >> > Cc: lars hofhansl >> > Sent: Thursday, December 31, 2015 6:10 AM >> > Subject: Re: Compact after manual flush broken in 1.0+? >> > >> > Indeed. I already informed the list about that months ago ;) >> > >> > 2015-12-31 9:09 GMT-05:00 Ted Yu : >> > >> > > bq. it looks like this is an oversight >> > > >> > > I agree. >> > > >> > > On Thu, Dec 31, 2015 at 12:04 AM, wrote: >> > > >> > > > Normally HBase triggers a compaction after a manual flush (of a >> region >> > or >> > > > table). >> > > > Looks like in 1.0 and later the code is still there but not actually >> > > > exercised, because FlushTableSubprocedure calls Region.flush() >> directly >> > > and >> > > > ignores the return code. >> > > > >> > > > Was that by design? If so, we can remove the flushResult logic. But >> it >> > > > looks like this is an oversight and we want to trigger the >> compaction >> > > still. >> > > > Comments? >> > > > -- Lars >> > > > >> > > > >> > > >> > >> > >> > >> > >> >
Re: Compact after manual flush broken in 1.0+?
>> Normally HBase triggers a compaction after a manual flush (of a region or table). Yes, I found this while working on FIFOCompaction test. Manual flush does not trigger compaction. But wait, we can manually trigger compaction if we need :). -Vlad On Wed, Jan 6, 2016 at 1:23 PM, Jean-Marc Spaggiari wrote: > Unable to find it. Feel free to open. > > JMS > > 2015-12-31 12:36 GMT-05:00 Jean-Marc Spaggiari : > > > I think there is one. I will search later today... > > Le 2015-12-31 11:35 AM, "Ted Yu" a écrit : > > > >> A brief search turned up no hit. > >> > >> Please go ahead with new JIRA. > >> > >> On Thu, Dec 31, 2015 at 8:32 AM, wrote: > >> > >> > Cool :) > >> > Is there a jira already? Otherwise I'll file one. > >> > -- Lars > >> > > >> > From: Jean-Marc Spaggiari > >> > To: dev > >> > Cc: lars hofhansl > >> > Sent: Thursday, December 31, 2015 6:10 AM > >> > Subject: Re: Compact after manual flush broken in 1.0+? > >> > > >> > Indeed. I already informed the list about that months ago ;) > >> > > >> > 2015-12-31 9:09 GMT-05:00 Ted Yu : > >> > > >> > > bq. it looks like this is an oversight > >> > > > >> > > I agree. > >> > > > >> > > On Thu, Dec 31, 2015 at 12:04 AM, wrote: > >> > > > >> > > > Normally HBase triggers a compaction after a manual flush (of a > >> region > >> > or > >> > > > table). > >> > > > Looks like in 1.0 and later the code is still there but not > actually > >> > > > exercised, because FlushTableSubprocedure calls Region.flush() > >> directly > >> > > and > >> > > > ignores the return code. > >> > > > > >> > > > Was that by design? If so, we can remove the flushResult logic. > But > >> it > >> > > > looks like this is an oversight and we want to trigger the > >> compaction > >> > > still. > >> > > > Comments? > >> > > > -- Lars > >> > > > > >> > > > > >> > > > >> > > >> > > >> > > >> > > >> > > >
[jira] [Created] (HBASE-15075) Allow region split request to carry metadata
Ted Yu created HBASE-15075: -- Summary: Allow region split request to carry metadata Key: HBASE-15075 URL: https://issues.apache.org/jira/browse/HBASE-15075 Project: HBase Issue Type: Improvement Reporter: Ted Yu During the process of improving region normalization feature, I found that if region split request triggered by the execution of SplitNormalizationPlan fails, there is no way of knowing whether the failed split originated from region normalization. The association of particular split request with outcome of split would give RegionNormalizer information so that it can make better normalization decisions in the subsequent invocations. One approach is to embed metadata in SplitRequest which gets passed through RegionStateTransitionContext when RegionServerServices#reportRegionStateTransition() is called. This way, RegionStateListener can be notified with the metadata (id of the requester). See discussion on dev mailing list http://search-hadoop.com/m/YGbbCXdkivihp2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15076) make the HRegion.getScanner(Scan scan, List additionalScanners, boolean copyCellsFromSharedMem) to be public
liu ming created HBASE-15076: Summary: make the HRegion.getScanner(Scan scan, List additionalScanners, boolean copyCellsFromSharedMem) to be public Key: HBASE-15076 URL: https://issues.apache.org/jira/browse/HBASE-15076 Project: HBase Issue Type: Improvement Components: regionserver Reporter: liu ming Assignee: liu ming HRegion method getScanner(Scan scan, List additionalScanners, boolean copyCellsFromSharedMem) is protected. In Apache Trafodion, we need to invoke this getScanner method from a coprocessor. Since it is protected, Trafodion must overload the HRegion class and overload this method into a public method. It will be good to make this method public. It is very useful when one needs to combine several scan result in a single scanner. thanks, Ming -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Parallel scanning of snapshots using hbase client api
Using the client api, we can scan the snapshot using a single scanner. We currently create an instance of TableSnapshotScanner providing it with a unique dir location per scanner. We are currently on cdh 5.4.5 and using the hbase 1.0.0-cdh5.4.5 api. In order to get desired throughput, we tried to increase the number of parallel scanners but raising the no. of scanner instances throws the following exception, the occurrence increases as the no. of scanners goes up. Can the client api support multiple scanners for a single snapshot? java.io.IOException: java.util.concurrent.ExecutionException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException): failed to create file /hbase/archive/data/default//cb794cfb7948ba8b1f4e73b690dfbfe5/L/.links-a04e1a5b2141445eb1b9e2429f1eced2/cb794cfb7948ba8b1f4e73b690dfbfe5. for DFSClient_NONMAPREDUCE_672650916_1 for client 10.10.2.90 because current leaseholder is trying to recreate file. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:3077) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2783) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2676) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2560) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:585) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.create(AuthorizationProviderProxyClientProtocol.java:110) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:395) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038) at org.apache.hadoop.hbase.util.ModifyRegionUtils.createRegions(ModifyRegionUtils.java:162) at org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.cloneHdfsRegions(RestoreSnapshotHelper.java:561) at org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:237) at org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.restoreHdfsRegions(RestoreSnapshotHelper.java:159) at org.apache.hadoop.hbase.snapshot.RestoreSnapshotHelper.copySnapshotForScanner(RestoreSnapshotHelper.java:812) at org.apache.hadoop.hbase.client.TableSnapshotScanner.init(TableSnapshotScanner.java:156) at org.apache.hadoop.hbase.client.TableSnapshotScanner.(TableSnapshotScanner.java:124) at org.apache.hadoop.hbase.client.TableSnapshotScanner.(TableSnapshotScanner.java:101) at net.connexity.aro.data.AudienceScanner.join(AudienceScanner.scala:68) at net.connexity.aro.actor.ScanActor.joinAudience(ScanActor.scala:190) at net.connexity.aro.actor.ScanActor$$anonfun$receive$1.applyOrElse(ScanActor.scala:90) at akka.actor.Actor$class.aroundReceive(Actor.scala:467) at net.connexity.aro.actor.ScanActor.aroundReceive(ScanActor.scala:36) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) at akka.actor.ActorCell.invoke(ActorCell.scala:487) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) at akka.dispatch.Mailbox.run(Mailbox.scala:220) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) -- View this message in context: http://apache-hbase.679495.n3.nabble.com/Parallel-scanning-of-snapshots-using-hbase-client-api-tp4077014.html Sent from the HBase Developer mailing list archive at Nabble.com.
Re: identifying source of region split request
Could we use Procedure V2 framework to do split? It seems to be a great work... 2016-01-07 4:35 GMT+08:00 Ted Yu : > bq. is splitting and subsequent merging a necessarily bad thing? > > Please note that merging may happen before the size of R1 (R2) drops to > zero. Meaning, there may be churn in normalization activity. > I would say that the normalization in the scenario below is premature. > > Cheers > > On Wed, Jan 6, 2016 at 11:41 AM, Mikhail Antonov > wrote: > > > Yeah, I see. Btw, in scenario you described, is splitting and subsequent > > merging a necessarily bad thing? If data expiration happens not that > often, > > having more evenly sized regions in between major collections isn't worth > > it? > > > > On Wed, Jan 6, 2016 at 10:23 AM, Ted Yu wrote: > > > > > bq. collect statistics to blacklist some RSs with high failure rate > > > > > > The metadata would help pinpoint the regions which consistently fail > > split > > > (merge) in the recent past. The failure could be due to corrupt > HFile(s) > > or > > > other reason. > > > Having the statistics would also help normalizer avoid the following > > > scenario: > > > > > > region R gets split into R1 and R2 > > > size of R1 and R2 decreases due to expiration of data > > > R1 and R2 get merged into R' > > > more data comes into R', resulting in split, again > > > > > > Cheers > > > > > > On Wed, Jan 6, 2016 at 10:14 AM, Mikhail Antonov > > > > wrote: > > > > > > > Adding this tracing information to know who initiated split is in > > general > > > > useful thing. Right now though I'm not sure I see how that would help > > to > > > > make better normalization decisions? Region split failure implies > > > > underlying FS issue? Any examples/ideas? > > > > > > > > Kind of..collect statistics to blacklist some RSs with high failure > > rate > > > > and don't attempt to split regions hosted there in future? > > > > > > > > On Tue, Jan 5, 2016 at 2:55 PM, Ted Yu wrote: > > > > > > > > > Hi, > > > > > I recently worked on improving region normalization feature. > > > > > > > > > > If region split request triggered by the execution of > > > > > SplitNormalizationPlan fails, there is no way of knowing whether > the > > > > failed > > > > > split originated from region normalization. > > > > > Such association would give RegionNormalizer information so that it > > can > > > > > make better normalization decisions in the subsequent invocations. > > > > > > > > > > One enhancement I can think of is to embed metadata in SplitRequest > > > which > > > > > gets passed through RegionStateTransitionContext when > > > > > RegionServerServices#reportRegionStateTransition() is called. > > > > > This way, RegionStateListener can be notified with the metadata (id > > of > > > > the > > > > > requester). > > > > > > > > > > Comment is welcome. > > > > > > > > > > > > > > > > > > > > > -- > > > > Thanks, > > > > Michael Antonov > > > > > > > > > > > > > > > -- > > Thanks, > > Michael Antonov > > >
Re: identifying source of region split request
at some point we can use proc-v2 for split/merge HBASE-14551/HBASE-14552 but I'm a bit behind the schedule with the framework. me and Stephen are working to get in that direction. but it is probably a month or two away Matteo On Wed, Jan 6, 2016 at 7:49 PM, Heng Chen wrote: > Could we use Procedure V2 framework to do split? > It seems to be a great work... > > 2016-01-07 4:35 GMT+08:00 Ted Yu : > > > bq. is splitting and subsequent merging a necessarily bad thing? > > > > Please note that merging may happen before the size of R1 (R2) drops to > > zero. Meaning, there may be churn in normalization activity. > > I would say that the normalization in the scenario below is premature. > > > > Cheers > > > > On Wed, Jan 6, 2016 at 11:41 AM, Mikhail Antonov > > wrote: > > > > > Yeah, I see. Btw, in scenario you described, is splitting and > subsequent > > > merging a necessarily bad thing? If data expiration happens not that > > often, > > > having more evenly sized regions in between major collections isn't > worth > > > it? > > > > > > On Wed, Jan 6, 2016 at 10:23 AM, Ted Yu wrote: > > > > > > > bq. collect statistics to blacklist some RSs with high failure rate > > > > > > > > The metadata would help pinpoint the regions which consistently fail > > > split > > > > (merge) in the recent past. The failure could be due to corrupt > > HFile(s) > > > or > > > > other reason. > > > > Having the statistics would also help normalizer avoid the following > > > > scenario: > > > > > > > > region R gets split into R1 and R2 > > > > size of R1 and R2 decreases due to expiration of data > > > > R1 and R2 get merged into R' > > > > more data comes into R', resulting in split, again > > > > > > > > Cheers > > > > > > > > On Wed, Jan 6, 2016 at 10:14 AM, Mikhail Antonov < > olorinb...@gmail.com > > > > > > > wrote: > > > > > > > > > Adding this tracing information to know who initiated split is in > > > general > > > > > useful thing. Right now though I'm not sure I see how that would > help > > > to > > > > > make better normalization decisions? Region split failure implies > > > > > underlying FS issue? Any examples/ideas? > > > > > > > > > > Kind of..collect statistics to blacklist some RSs with high failure > > > rate > > > > > and don't attempt to split regions hosted there in future? > > > > > > > > > > On Tue, Jan 5, 2016 at 2:55 PM, Ted Yu > wrote: > > > > > > > > > > > Hi, > > > > > > I recently worked on improving region normalization feature. > > > > > > > > > > > > If region split request triggered by the execution of > > > > > > SplitNormalizationPlan fails, there is no way of knowing whether > > the > > > > > failed > > > > > > split originated from region normalization. > > > > > > Such association would give RegionNormalizer information so that > it > > > can > > > > > > make better normalization decisions in the subsequent > invocations. > > > > > > > > > > > > One enhancement I can think of is to embed metadata in > SplitRequest > > > > which > > > > > > gets passed through RegionStateTransitionContext when > > > > > > RegionServerServices#reportRegionStateTransition() is called. > > > > > > This way, RegionStateListener can be notified with the metadata > (id > > > of > > > > > the > > > > > > requester). > > > > > > > > > > > > Comment is welcome. > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Thanks, > > > > > Michael Antonov > > > > > > > > > > > > > > > > > > > > > -- > > > Thanks, > > > Michael Antonov > > > > > >
