[jira] [Reopened] (HBASE-16932) Implement small scan
[ https://issues.apache.org/jira/browse/HBASE-16932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reopened HBASE-16932: --- Reopen for fixing the findbugs issue. https://builds.apache.org/job/PreCommit-HBASE-Build/4219/artifact/patchprocess/branch-findbugs-hbase-client-warnings.html Seems the precommit job here didn't find out the issue? > Implement small scan > > > Key: HBASE-16932 > URL: https://issues.apache.org/jira/browse/HBASE-16932 > Project: HBase > Issue Type: Sub-task > Components: Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16932-v1.patch, HBASE-16932.patch > > > As said in HBASE-16838, we should have a scan method that returns a > CompletaFuture for the whole result. It is suitable for a small scan. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-16932) Implement small scan
[ https://issues.apache.org/jira/browse/HBASE-16932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-16932. --- Resolution: Fixed Oh, it is introduced in HBASE-16886. > Implement small scan > > > Key: HBASE-16932 > URL: https://issues.apache.org/jira/browse/HBASE-16932 > Project: HBase > Issue Type: Sub-task > Components: Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16932-v1.patch, HBASE-16932.patch > > > As said in HBASE-16838, we should have a scan method that returns a > CompletaFuture for the whole result. It is suitable for a small scan. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-16932) Implement small scan
[ https://issues.apache.org/jira/browse/HBASE-16932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reopened HBASE-16932: --- > Implement small scan > > > Key: HBASE-16932 > URL: https://issues.apache.org/jira/browse/HBASE-16932 > Project: HBase > Issue Type: Sub-task > Components: Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16932-v1.patch, HBASE-16932.patch > > > As said in HBASE-16838, we should have a scan method that returns a > CompletaFuture for the whole result. It is suitable for a small scan. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-16932) Implement small scan
[ https://issues.apache.org/jira/browse/HBASE-16932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-16932. --- Resolution: Fixed Pushed the addendum to master. Thanks [~tedyu]. > Implement small scan > > > Key: HBASE-16932 > URL: https://issues.apache.org/jira/browse/HBASE-16932 > Project: HBase > Issue Type: Sub-task > Components: Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16932-addendum.patch, HBASE-16932-v1.patch, > HBASE-16932.patch > > > As said in HBASE-16838, we should have a scan method that returns a > CompletaFuture for the whole result. It is suitable for a small scan. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Fixed: HBase Generate Website
Build status: Fixed
If successful, the website and docs have been generated. To update the live
site, follow the instructions below. If failed, skip to the bottom of this
email.
Use the following commands to download the patch and apply it to a clean branch
based on origin/asf-site. If you prefer to keep the hbase-site repo around
permanently, you can skip the clone step.
git clone https://git-wip-us.apache.org/repos/asf/hbase-site.git
cd hbase-site
wget -O-
https://builds.apache.org/job/hbase_generate_website/391/artifact/website.patch.zip
| funzip > 9607409fca41edf9484876ddff8bbd6ee9becae5.patch
git fetch
git checkout -b asf-site-9607409fca41edf9484876ddff8bbd6ee9becae5
origin/asf-site
git am --whitespace=fix 9607409fca41edf9484876ddff8bbd6ee9becae5.patch
At this point, you can preview the changes by opening index.html or any of the
other HTML pages in your local
asf-site-9607409fca41edf9484876ddff8bbd6ee9becae5 branch.
There are lots of spurious changes, such as timestamps and CSS styles in
tables, so a generic git diff is not very useful. To see a list of files that
have been added, deleted, renamed, changed type, or are otherwise interesting,
use the following command:
git diff --name-status --diff-filter=ADCRTXUB origin/asf-site
To see only files that had 100 or more lines changed:
git diff --stat origin/asf-site | grep -E '[1-9][0-9]{2,}'
When you are satisfied, publish your changes to origin/asf-site using these
commands:
git commit --allow-empty -m "Empty commit" # to work around a current ASF
INFRA bug
git push origin asf-site-9607409fca41edf9484876ddff8bbd6ee9becae5:asf-site
git checkout asf-site
git branch -D asf-site-9607409fca41edf9484876ddff8bbd6ee9becae5
Changes take a couple of minutes to be propagated. You can verify whether they
have been propagated by looking at the Last Published date at the bottom of
http://hbase.apache.org/. It should match the date in the index.html on the
asf-site branch in Git.
As a courtesy- reply-all to this email to let other committers know you pushed
the site.
If failed, see https://builds.apache.org/job/hbase_generate_website/391/console
[DISCUSS] FileSystem Quotas in HBase
Hi folks, I'd like to propose the introduction of FileSystem quotas to HBase. Here's a design doc[1] available which (hopefully) covers all of the salient points of what I think an initial version of such a feature would include. tl;dr We can define quotas on tables and namespaces. Region size is computed by RegionServers and sent to the Master. The Master inspects the sizes of Regions, rolling up to table and namespace sizes. Defined quotas in the quota table are evaluated given the computed sizes, and, for those tables/namespaces violating the quota, RegionServers are informed to take some action to limit any further filesystem growth by that table/namespace. I'd encourage you to give the document a read -- I tried to cover as much as I could without getting unnecessarily bogged down in implementation details. Feedback is, of course, welcomed. I'd like to start sketching out a breakdown of the work (all writing and no programming makes Josh a sad boy). I'm happy to field any/all questions. Thanks in advance. - Josh [1] http://home.apache.org/~elserj/hbase/FileSystemQuotasforApacheHBase.pdf
Re: [DISCUSS] FileSystem Quotas in HBase
Josh: Can you put the doc in google doc so that people can comment on it ? Is there a JIRA opened for this work ? Please open one if there is none. Thanks On Fri, Oct 28, 2016 at 9:00 AM, Josh Elser wrote: > Hi folks, > > I'd like to propose the introduction of FileSystem quotas to HBase. > > Here's a design doc[1] available which (hopefully) covers all of the > salient points of what I think an initial version of such a feature would > include. > > tl;dr We can define quotas on tables and namespaces. Region size is > computed by RegionServers and sent to the Master. The Master inspects the > sizes of Regions, rolling up to table and namespace sizes. Defined quotas > in the quota table are evaluated given the computed sizes, and, for those > tables/namespaces violating the quota, RegionServers are informed to take > some action to limit any further filesystem growth by that table/namespace. > > I'd encourage you to give the document a read -- I tried to cover as much > as I could without getting unnecessarily bogged down in implementation > details. > > Feedback is, of course, welcomed. I'd like to start sketching out a > breakdown of the work (all writing and no programming makes Josh a sad > boy). I'm happy to field any/all questions. Thanks in advance. > > - Josh > > [1] http://home.apache.org/~elserj/hbase/FileSystemQuotasforApac > heHBase.pdf >
Re: [DISCUSS] FileSystem Quotas in HBase
Sure thing, Ted. https://docs.google.com/document/d/1VtLWDkB2tpwc_zgCNPE1ulZOeecF-YA2FYSK3TSs_bw/edit?usp=sharing Let me open an umbrella issue for now. I can break up the work later. https://issues.apache.org/jira/browse/HBASE-16961 Ted Yu wrote: Josh: Can you put the doc in google doc so that people can comment on it ? Is there a JIRA opened for this work ? Please open one if there is none. Thanks On Fri, Oct 28, 2016 at 9:00 AM, Josh Elser wrote: Hi folks, I'd like to propose the introduction of FileSystem quotas to HBase. Here's a design doc[1] available which (hopefully) covers all of the salient points of what I think an initial version of such a feature would include. tl;dr We can define quotas on tables and namespaces. Region size is computed by RegionServers and sent to the Master. The Master inspects the sizes of Regions, rolling up to table and namespace sizes. Defined quotas in the quota table are evaluated given the computed sizes, and, for those tables/namespaces violating the quota, RegionServers are informed to take some action to limit any further filesystem growth by that table/namespace. I'd encourage you to give the document a read -- I tried to cover as much as I could without getting unnecessarily bogged down in implementation details. Feedback is, of course, welcomed. I'd like to start sketching out a breakdown of the work (all writing and no programming makes Josh a sad boy). I'm happy to field any/all questions. Thanks in advance. - Josh [1] http://home.apache.org/~elserj/hbase/FileSystemQuotasforApac heHBase.pdf
[jira] [Created] (HBASE-16961) FileSystem Quotas
Josh Elser created HBASE-16961: -- Summary: FileSystem Quotas Key: HBASE-16961 URL: https://issues.apache.org/jira/browse/HBASE-16961 Project: HBase Issue Type: New Feature Reporter: Josh Elser Assignee: Josh Elser Umbrella issue for tracking the filesystem utilization of HBase data, defining quotas on that utilization, and enforcement when utilization exceeds the limits of the quota. At a high level: we can define quotas on tables and namespaces. Region size is computed by RegionServers and sent to the Master. The Master inspects the sizes of Regions, rolling up to table and namespace sizes. Defined quotas in the quota table are evaluated given the computed sizes, and, for those tables/namespaces violating the quota, RegionServers are informed to take some action to limit any further filesystem growth by that table/namespace. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16962) Add readPoint to preCompactScannerOpen() API
Thiruvel Thirumoolan created HBASE-16962: Summary: Add readPoint to preCompactScannerOpen() API Key: HBASE-16962 URL: https://issues.apache.org/jira/browse/HBASE-16962 Project: HBase Issue Type: Bug Affects Versions: 2.0.0 Reporter: Thiruvel Thirumoolan Assignee: Thiruvel Thirumoolan Similar to HBASE-15759, I would like to add readPoint to the preCompactScannerOpen() API. I have a CP where I create a StoreScanner() as part of the preCompactScannerOpen() API. I need the readpoint which was obtained in Compactor.compact() method to be consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16963) Fixup our hbase-personality post YETUS-322 flip verify_ result codes
stack created HBASE-16963: - Summary: Fixup our hbase-personality post YETUS-322 flip verify_ result codes Key: HBASE-16963 URL: https://issues.apache.org/jira/browse/HBASE-16963 Project: HBase Issue Type: Sub-task Reporter: stack -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16964) Successfully archived files are not cleared from compacted store file list if archiving of any file fails
Gary Helmling created HBASE-16964: - Summary: Successfully archived files are not cleared from compacted store file list if archiving of any file fails Key: HBASE-16964 URL: https://issues.apache.org/jira/browse/HBASE-16964 Project: HBase Issue Type: Bug Components: regionserver Reporter: Gary Helmling Assignee: Gary Helmling Priority: Blocker Fix For: 1.3.0 In HStore.removeCompactedFiles(), we only clear archived files from StoreFileManager's list of compactedfiles if _all_ files were archived successfully. If we encounter an error archiving any of the files, then any files which were already archived will remain in the list of compactedfiles. Even worse, this means that all subsequent attempts to archive the list of compacted files will fail (as the previously successfully archived files still in the list will now throw FileNotFoundException), and the list of compactedfiles will never be cleared from that point on. Finally, when the region closes, we will again throw an exception out of HStore.removeCompactedFiles(), in this case causing a regionserver abort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-16951) 1.3 assembly scripts dont package hbase-archetypes in the tarball
[ https://issues.apache.org/jira/browse/HBASE-16951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov resolved HBASE-16951. - Resolution: Fixed > 1.3 assembly scripts dont package hbase-archetypes in the tarball > - > > Key: HBASE-16951 > URL: https://issues.apache.org/jira/browse/HBASE-16951 > Project: HBase > Issue Type: Sub-task > Components: build >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Fix For: 1.3.0 > > Attachments: HBASE-16951.branch-1.3.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16965) Update hbase personality after YETUS-322 "flip verify_ result codes"
stack created HBASE-16965: - Summary: Update hbase personality after YETUS-322 "flip verify_ result codes" Key: HBASE-16965 URL: https://issues.apache.org/jira/browse/HBASE-16965 Project: HBase Issue Type: Bug Reporter: stack HBase personality makes use of verify_needed_test Needs to use the new idiom around verify_needed_test result post YETUS-322. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16966) Re-enable TestSimpleRpcScheduler#testCoDelScheduling in master branch
Mikhail Antonov created HBASE-16966: --- Summary: Re-enable TestSimpleRpcScheduler#testCoDelScheduling in master branch Key: HBASE-16966 URL: https://issues.apache.org/jira/browse/HBASE-16966 Project: HBase Issue Type: Bug Components: test Affects Versions: 2.0.0 Reporter: Mikhail Antonov Fix For: 2.0.0 see HBASE-16743. Creating separate jira so we don't forget. This test been flaky for a while. Need to make sure it's fixed. it's already enabled on branch-1 and branch-1.3 (rather, never was disabled) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-16743) TestSimpleRpcScheduler#testCoDelScheduling is broke
[
https://issues.apache.org/jira/browse/HBASE-16743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov resolved HBASE-16743.
-
Resolution: Fixed
marking resolved, and filed a separate jira to re-enable it on master
> TestSimpleRpcScheduler#testCoDelScheduling is broke
> ---
>
> Key: HBASE-16743
> URL: https://issues.apache.org/jira/browse/HBASE-16743
> Project: HBase
> Issue Type: Bug
> Components: rpc
>Affects Versions: 1.3.0
>Reporter: stack
>Assignee: Mikhail Antonov
> Fix For: 1.3.0
>
> Attachments: HBASE-16743.branch-1.patch
>
>
> The testCoDelScheduling test is broke. Here are some notes on it. I have
> disabled it in the HBASE-15638 shading patch.
> {code}
> I don't get this test. When I time this test, the minDelay is > 2 * codel
> delay from the get go. So we are always overloaded. The test below would seem
> to complete the queuing of all the CallRunners inside the codel check
> interval. I don't think we are skipping codel checking. Second, I think this
> test has been broken since HBASE-16089 Add on FastPath for CoDel went in.
> The thread name we were looking for was the name BEFORE we updated: i.e.
> "RpcServer.CodelBQ.default.handler". But same patch changed the name of the
> codel fastpath thread to: new
> FastPathBalancedQueueRpcExecutor("CodelFPBQ.default", handlerCount,
> numCallQueues...
> Codel is hard to test. This test is going to be flakey given it all
> timer-based. Disabling for now till chat
> {code}
> FYI [~mantonov]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16967) Excluded shaded java classes from findbugs check
stack created HBASE-16967: - Summary: Excluded shaded java classes from findbugs check Key: HBASE-16967 URL: https://issues.apache.org/jira/browse/HBASE-16967 Project: HBase Issue Type: Sub-task Reporter: stack Carry over from parent issue. [~Apache9] notes that we get findbugs warnings in hbase-protocol-shaded when a build in this module. The findbugs complaints are about google protobuf that we've relocated and checked in here. Exclude. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [VOTE] Second release candidate for HBase 1.2.4 is available
+1 Checked signature (lets sign each other's keys Sean) and hash. Untarred. Checked layout. Ran apache rat. Built with apache-release profile. Built site. Poked around. Looks right. Started up in single mode, loaded data, verified it loaded. Checked logs (We need to fix the zk spew in our log). Shut it down. S On Tue, Oct 25, 2016 at 8:17 PM, Sean Busbey wrote: > The second release candidate for HBase 1.2.4 is available for download at: > > https://dist.apache.org/repos/dist/dev/hbase/hbase-1.2.4RC1/ > > Maven artifacts are also available in a staging repository at: > > https://repository.apache.org/content/repositories/orgapachehbase-1156/ > > Artifacts are signed with my key (0D80DB7C) published in our KEYS > file at http://www.apache.org/dist/hbase/KEYS > > The RC corresponds to the signed tag 1.2.4RC1, which currently points > to commit ref > > 67592f3d062743907f8c5ae00dbbe1ae4f69e5af > > The detailed source and binary compatibility report vs 1.2.3 has been > published for your review, at: > > https://dist.apache.org/repos/dist/dev/hbase/hbase-1.2.4RC1/ > 1.2.3_1.2.4RC1_compat_report.html > > HBase 1.2.4 is the fourth patch release in the HBase 1.2 line, continuing > on > the theme of bringing a stable, reliable database to the Hadoop and NoSQL > communities. This release includes about 30 bug fixes since 1.2.3. > Critical fixes include: > > * HBASE-16340 Xerces implementation jars removed (incompatible change) > * HBASE-15984 Given failure to parse a given WAL that was closed > cleanly, replay the WAL. > * HBASE-16931 Setting cell's seqId to zero in compaction flow might > cause RS down. > * HBASE-16721 Concurrency issue in WAL unflushed seqId tracking > > The full list of fixes included in this release is available at: > > https://issues.apache.org/jira/secure/ReleaseNote.jspa? > projectId=12310753&version=12338116 > > and in the CHANGES.txt file included in the distribution. > > Please try out this candidate and vote +1/-1 on whether we should > release these artifacts as HBase 1.2.4. > > The VOTE will remain open for atleast 72 hours. Given sufficient votes > I would like to close it on November 1st, 2016. > > thanks! >
State of 1.3 first RC
Guys, I've been working on getting the first RC out this week and want to share an update on the current state of the things. - HBASE-16951 - was found and fixed, when we don't include hbase-archetypes in the tarball assembly - HBASE-16570 - optimization that caused a regression in the balancer, it was reverted from 1.3. Once fixed that would go to 1.4 and perhaps 1.3.1. - HBASE-16743 - flaky TestSimpleRpcScheduler, fixed now (I ran several cycles of 50 iterations each and all have passed). Now, there's presently one remaining blocker - HBASE-16964, when we don't properly handle archiving of store files after compactions (thanks Gary for digging into that!). See the jira for details. Also, as a note - the above is a third major bug around compactions that we saw on 1.3 during our testing last month (see also HBASE-16788 and HBASE-16754).. So during the upcoming RC testing let's try to stress this area. Until 1.3 is out let's consider branch-1.3 to be in a lockdown for new changes. Ping me if you think there is something that I'm missing that should go out in 1.3. Otherwise, important but non-blocker fixes in flight would go in 1.3.1. Stay tuned! Mikhail
[jira] [Created] (HBASE-16968) Refactor FanOutOneBlockAsyncDFSOutput
Duo Zhang created HBASE-16968: - Summary: Refactor FanOutOneBlockAsyncDFSOutput Key: HBASE-16968 URL: https://issues.apache.org/jira/browse/HBASE-16968 Project: HBase Issue Type: Sub-task Components: io, wal Affects Versions: 2.0.0 Reporter: Duo Zhang Fix For: 2.0.0 As we have moved to jdk8, and dropped the support of hadoop-2.4.x and hadoop-2.5.x on master, we could use CompletableFuture instead of CompletionHandler to make the code more clear, and also remove the support of hadoop-2.4.x and hadoop-2.5.x to remove lots of the reflection code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-3616) Add per region request information to HServerLoad
[
https://issues.apache.org/jira/browse/HBASE-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HBASE-3616.
---
Resolution: Won't Fix
Gone stale.
> Add per region request information to HServerLoad
> -
>
> Key: HBASE-3616
> URL: https://issues.apache.org/jira/browse/HBASE-3616
> Project: HBase
> Issue Type: Improvement
> Components: master
>Reporter: Ted Yu
>
> HBASE-3507 added per region request count.
> We should utilize this information so that HServerLoad can provide moving
> average of request counts to load balancer.
> We can update this method in HRegionServer:
> {code}
> private HServerLoad buildServerLoad() {
> {code}
> The above method can aggregate request counts from HRegions and store it in
> HServerLoad.RegionLoad
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
