[jira] [Updated] (HBASE-17326) Fix findbugs warning in BufferedMutatorParams
[ https://issues.apache.org/jira/browse/HBASE-17326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17326: -- Assignee: stack Status: Patch Available (was: Open) Fix for two findbugs warnings in master > Fix findbugs warning in BufferedMutatorParams > - > > Key: HBASE-17326 > URL: https://issues.apache.org/jira/browse/HBASE-17326 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang >Assignee: stack > Attachments: HBASE-17326.master.001.patch > > > https://builds.apache.org/job/PreCommit-HBASE-Build/4947/artifact/patchprocess/branch-findbugs-hbase-client-warnings.html > org.apache.hadoop.hbase.client.BufferedMutatorParams defines clone() but > doesn't implement Cloneable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17326) Fix findbugs warning in BufferedMutatorParams
[ https://issues.apache.org/jira/browse/HBASE-17326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17326: -- Attachment: HBASE-17326.master.001.patch > Fix findbugs warning in BufferedMutatorParams > - > > Key: HBASE-17326 > URL: https://issues.apache.org/jira/browse/HBASE-17326 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang > Attachments: HBASE-17326.master.001.patch > > > https://builds.apache.org/job/PreCommit-HBASE-Build/4947/artifact/patchprocess/branch-findbugs-hbase-client-warnings.html > org.apache.hadoop.hbase.client.BufferedMutatorParams defines clone() but > doesn't implement Cloneable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16859) Use Bytebuffer pool for non java clients specifically for scans/gets
[ https://issues.apache.org/jira/browse/HBASE-16859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755097#comment-15755097 ] stack commented on HBASE-16859: --- Any downside [~ram_krish]? > Use Bytebuffer pool for non java clients specifically for scans/gets > > > Key: HBASE-16859 > URL: https://issues.apache.org/jira/browse/HBASE-16859 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-16859_V1.patch, HBASE-16859_V2.patch, > HBASE-16859_V2.patch, HBASE-16859_V4.patch, HBASE-16859_V5.patch > > > In case of non java clients we still write the results and header into a on > demand byte[]. This can be changed to use the BBPool (onheap or offheap > buffer?). > But the basic problem is to identify if the response is for scans/gets. > - One easy way to do it is use the MethodDescriptor per Call and use the > name of the MethodDescriptor to identify it is a scan/get. But this will > pollute RpcServer by checking for scan/get type response. > - Other way is always set the result to cellScanner but we know that > isClientCellBlockSupported is going to false for non PB clients. So ignore > the cellscanner and go ahead with the results in PB. But this is not clean > - third one is that we already have a RpccallContext being passed to the RS. > In case of scan/gets/multiGets we already set a Rpccallback for shipped call. > So here on response we can check if the callback is not null and check for > isclientBlockSupported. In this case we can get the BB from the pool and > write the result and header to that BB. May be this looks clean? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15248) BLOCKSIZE 4k should result in 4096 bytes on disk; i.e. fit inside a BucketCache 'block' of 4k
[ https://issues.apache.org/jira/browse/HBASE-15248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755063#comment-15755063 ] stack commented on HBASE-15248: --- bq. So right now u have removed the eager fetch of next block header right? No. I did not. On study, it would be too disruptive removing the fetch of the next block header: HBASE-17185 "Purge the seek of the next block reading HFileBlocks" On this issue, the way we do sizing now -- i.e. we write bigger than what is configured -- is confusing. This issue should be about rationalizing this. > BLOCKSIZE 4k should result in 4096 bytes on disk; i.e. fit inside a > BucketCache 'block' of 4k > - > > Key: HBASE-15248 > URL: https://issues.apache.org/jira/browse/HBASE-15248 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack > > Chatting w/ a gentleman named Daniel Pol who is messing w/ bucketcache, he > wants blocks to be the size specified in the configuration and no bigger. His > hardware set ups fetches pages of 4k and so a block that has 4k of payload > but has then a header and the header of the next block (which helps figure > whats next when scanning) ends up being 4203 bytes or something, and this > then then translates into two seeks per block fetch. > This issue is about what it would take to stay inside our configured size > boundary writing out blocks. > If not possible, give back better signal on what to do so you could fit > inside a particular constraint. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15432) TableInputFormat - support multiple column families scan
[
https://issues.apache.org/jira/browse/HBASE-15432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15755066#comment-15755066
]
Hadoop QA commented on HBASE-15432:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 44s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
1s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 23s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 93m 17s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 133m 35s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843608/HBASE-15432.master.003.patch
|
| JIRA Issue | HBASE-15432 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 4dc8d7593dfe 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 6acbee1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4950/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4950/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> TableInputFormat - support multiple column families scan
>
>
> Key: HBASE-15432
> URL: https://issues.apache.org/jira/browse/HBASE-15432
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 2.0.0, 0.98.16, 0.98.16.1
>Reporter: nirav patel
>Assignee: Xuesen Liang
> Fix For: 2.0.0
>
> Attachments:
[jira] [Updated] (HBASE-17292) Add observer notification before bulk loaded hfile is moved to region directory
[ https://issues.apache.org/jira/browse/HBASE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17292: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Add observer notification before bulk loaded hfile is moved to region > directory > --- > > Key: HBASE-17292 > URL: https://issues.apache.org/jira/browse/HBASE-17292 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 17292.v1.txt, 17292.v2.txt, 17292.v3.txt > > > Currently the postBulkLoadHFile() hook notifies the locations of bulk loaded > hfiles. > However, if bulk load fails after hfile is moved to region directory but > before postBulkLoadHFile() hook is called, there is no way for pluggable > components (replication - see HBASE-17290, backup / restore) to know which > hfile(s) have been moved to region directory. > Even if postBulkLoadHFile() is called in finally block, the write (to backup > table or zookeeper) issued from postBulkLoadHFile() may fail, ending up with > same situation. > This issue adds a preCommitStoreFile() hook which notifies path of to be > committed hfile before bulk loaded hfile is moved to region directory. > With preCommitStoreFile() hook, write (to backup table or zookeeper) can be > issued before the movement of hfile. > If write fails, IOException would make bulk load fail, not leaving hfile in > region directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17254) Do alignment in heapOverhead() for ExtendedCell implementations
[ https://issues.apache.org/jira/browse/HBASE-17254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754793#comment-15754793 ] Anoop Sam John commented on HBASE-17254: NP.. Take ur time... BTW, I have changed the issue to be a minor one.. Not so critical any way. > Do alignment in heapOverhead() for ExtendedCell implementations > --- > > Key: HBASE-17254 > URL: https://issues.apache.org/jira/browse/HBASE-17254 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Xiang Li >Assignee: Xiang Li >Priority: Minor > > In KeyValue (and maybe other ExtendedCell implementations), heapOverhead() > does not do alignment(padding gap). I think alignment is required when > calculating heapOverhead, because the space losses due to alignment can not > be used by other objects in heap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17254) Do alignment in heapOverhead() for ExtendedCell implementations
[ https://issues.apache.org/jira/browse/HBASE-17254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-17254: --- Priority: Minor (was: Critical) > Do alignment in heapOverhead() for ExtendedCell implementations > --- > > Key: HBASE-17254 > URL: https://issues.apache.org/jira/browse/HBASE-17254 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Xiang Li >Assignee: Xiang Li >Priority: Minor > > In KeyValue (and maybe other ExtendedCell implementations), heapOverhead() > does not do alignment(padding gap). I think alignment is required when > calculating heapOverhead, because the space losses due to alignment can not > be used by other objects in heap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15432) TableInputFormat - support multiple column families scan
[ https://issues.apache.org/jira/browse/HBASE-15432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuesen Liang updated HBASE-15432: - Attachment: HBASE-15432.master.003.patch fix CellCounter > TableInputFormat - support multiple column families scan > > > Key: HBASE-15432 > URL: https://issues.apache.org/jira/browse/HBASE-15432 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0, 0.98.16, 0.98.16.1 >Reporter: nirav patel >Assignee: Xuesen Liang > Fix For: 2.0.0 > > Attachments: HBASE-15432.master.002.patch, > HBASE-15432.master.003.patch, HBASE-15432.master.patch > > > Currently Hbase TableInputFormat class has SCAN_COLUMN_FAMILY and > SCAN_COLUMNS. SCAN_COLUMN_FAMILY can only scan single column family. If we > need to scan multiple column families from a table then we must use > SCAN_COLUMNS where we must provide both columns and column families which is > not a convenient. Can we have a SCAN_COLUMN_FAMILIES which supports scan of > multiple column families. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17319) Truncate table with preserve after split may cause truncation to fail
[
https://issues.apache.org/jira/browse/HBASE-17319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754655#comment-15754655
]
Ted Yu commented on HBASE-17319:
Please see what may have caused TestTruncateTableProcedure to hang.
For every bug you fix in branch-1, check whether master branch has the same
issue. If so, understand the difference in master and fix the bug as well.
> Truncate table with preserve after split may cause truncation to fail
> -
>
> Key: HBASE-17319
> URL: https://issues.apache.org/jira/browse/HBASE-17319
> Project: HBase
> Issue Type: Bug
> Components: Admin
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Allan Yang
>Assignee: Allan Yang
> Fix For: 1.4.0
>
> Attachments: HBASE-17319-branch-1.patch, HBASE-17319.patch
>
>
> In truncateTableProcedue , when getting tables regions from meta to recreate
> new regions, split parents are not excluded, so the new regions can end up
> with the same start key, and the same region dir:
> {noformat}
> 2016-12-14 20:15:22,231 WARN [RegionOpenAndInitThread-writetest-1]
> regionserver.HRegionFileSystem: Trying to create a region that already exists
> on disk:
> hdfs://hbasedev1/zhengyan-hbase11-func2/.tmp/data/default/writetest/9b2c8d1539cd92661703ceb8a4d518a1
> {noformat}
> The truncateTableProcedue will retry forever and never get success.
> A attached unit test shows everything.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17282) Reduce the redundant requests to meta table
[ https://issues.apache.org/jira/browse/HBASE-17282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754380#comment-15754380 ] Duo Zhang commented on HBASE-17282: --- Any concerns? [~carp84] [~stack]. Thanks. > Reduce the redundant requests to meta table > --- > > Key: HBASE-17282 > URL: https://issues.apache.org/jira/browse/HBASE-17282 > Project: HBase > Issue Type: Sub-task > Components: Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17282-v1.patch, HBASE-17282.patch > > > This usually happens at startup when the meta cache is empty. There will be a > lot of locating requests, but most of will have same results. Things become > worse if we do batch operations with AsyncTable as we will send a locating > request for each operation concurrently. > We need to reduce the redundant requests to meta table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17254) Do alignment in heapOverhead() for ExtendedCell implementations
[ https://issues.apache.org/jira/browse/HBASE-17254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754267#comment-15754267 ] Xiang Li commented on HBASE-17254: -- Hi [~anoop.hbase], just would like to let you know that I did not stop working on this JIRA but is reading HBASE-15950 and learning MSLAB related JIRA/knowledge > Do alignment in heapOverhead() for ExtendedCell implementations > --- > > Key: HBASE-17254 > URL: https://issues.apache.org/jira/browse/HBASE-17254 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Xiang Li >Assignee: Xiang Li >Priority: Critical > > In KeyValue (and maybe other ExtendedCell implementations), heapOverhead() > does not do alignment(padding gap). I think alignment is required when > calculating heapOverhead, because the space losses due to alignment can not > be used by other objects in heap. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17326) Fix findbugs warning in BufferedMutatorParams
[ https://issues.apache.org/jira/browse/HBASE-17326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-17326: --- Affects Version/s: 2.0.0 > Fix findbugs warning in BufferedMutatorParams > - > > Key: HBASE-17326 > URL: https://issues.apache.org/jira/browse/HBASE-17326 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang > > https://builds.apache.org/job/PreCommit-HBASE-Build/4947/artifact/patchprocess/branch-findbugs-hbase-client-warnings.html > org.apache.hadoop.hbase.client.BufferedMutatorParams defines clone() but > doesn't implement Cloneable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17326) Fix findbugs warning in BufferedMutatorParams
Guanghao Zhang created HBASE-17326: -- Summary: Fix findbugs warning in BufferedMutatorParams Key: HBASE-17326 URL: https://issues.apache.org/jira/browse/HBASE-17326 Project: HBase Issue Type: Bug Reporter: Guanghao Zhang https://builds.apache.org/job/PreCommit-HBASE-Build/4947/artifact/patchprocess/branch-findbugs-hbase-client-warnings.html org.apache.hadoop.hbase.client.BufferedMutatorParams defines clone() but doesn't implement Cloneable -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17174) Refactor the AsyncProcess
[ https://issues.apache.org/jira/browse/HBASE-17174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-17174: -- Status: Patch Available (was: Open) > Refactor the AsyncProcess > - > > Key: HBASE-17174 > URL: https://issues.apache.org/jira/browse/HBASE-17174 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17174.v0.patch, HBASE-17174.v1.patch, > HBASE-17174.v10.patch, HBASE-17174.v11.patch, HBASE-17174.v2.patch, > HBASE-17174.v3.patch, HBASE-17174.v4.patch, HBASE-17174.v5.patch, > HBASE-17174.v6.patch, HBASE-17174.v7.patch, HBASE-17174.v8.patch, > HBASE-17174.v9.patch > > > The following are reasons of reusing the pool and AP. > # A update-heavy application, for example, loader, creates many > BufferedMutator for batch updates. But these BufferedMutators can’t share a > large threadpool because the shutdown() method will be called when closing > any BufferedMutator. This patch adds a flag into BufferedMutatorParams for > preventing calling the shutdown() method in BufferedMutatorImpl#close > # The AsyncProcess has the powerful traffic control, but the control is > against the single table currently. We should allow alternate traffic control > implementation for advanced user who want more control. > All suggestions are welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17174) Refactor the AsyncProcess
[ https://issues.apache.org/jira/browse/HBASE-17174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-17174: -- Attachment: HBASE-17174.v11.patch v11 allows allow alternate traffic control implementation so that we don't expose the AP. > Refactor the AsyncProcess > - > > Key: HBASE-17174 > URL: https://issues.apache.org/jira/browse/HBASE-17174 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17174.v0.patch, HBASE-17174.v1.patch, > HBASE-17174.v10.patch, HBASE-17174.v11.patch, HBASE-17174.v2.patch, > HBASE-17174.v3.patch, HBASE-17174.v4.patch, HBASE-17174.v5.patch, > HBASE-17174.v6.patch, HBASE-17174.v7.patch, HBASE-17174.v8.patch, > HBASE-17174.v9.patch > > > The following are reasons of reusing the pool and AP. > # A update-heavy application, for example, loader, creates many > BufferedMutator for batch updates. But these BufferedMutators can’t share a > large threadpool because the shutdown() method will be called when closing > any BufferedMutator. This patch adds a flag into BufferedMutatorParams for > preventing calling the shutdown() method in BufferedMutatorImpl#close > # The AsyncProcess has the powerful traffic control, but the control is > against the single table currently. We should allow alternate traffic control > implementation for advanced user who want more control. > All suggestions are welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11392) add/remove peer requests should be routed through master
[
https://issues.apache.org/jira/browse/HBASE-11392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754063#comment-15754063
]
Hadoop QA commented on HBASE-11392:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 49s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
53s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 21s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 10m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 56s
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 50s
{color} | {color:red} hbase-client in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 10m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 15 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 21s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 27s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 57s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 93m 32s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 170m 25s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843564/HBASE-11392-v3.patch |
| JIRA Issue | HBASE-11392 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile cc hbaseprotoc |
| uname | Linux 0fba52ffcd9f 3.13.0-103-generic
[jira] [Commented] (HBASE-17290) Potential loss of data for replication of bulk loaded hfiles
[ https://issues.apache.org/jira/browse/HBASE-17290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754019#comment-15754019 ] Anoop Sam John commented on HBASE-17290: I believe this point was discussed in the original feature issue.. Sorry I totally forgot abt it now.. [~ashish singhi] on vacation right now and once he is back, he can comment abt the discussion points happened at that time. > Potential loss of data for replication of bulk loaded hfiles > > > Key: HBASE-17290 > URL: https://issues.apache.org/jira/browse/HBASE-17290 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu > > Currently the support for replication of bulk loaded hfiles relies on bulk > load marker written in the WAL. > The move of bulk loaded hfile(s) (into region directory) may succeed but the > write of bulk load marker may fail. > This means that although bulk loaded hfile is being served in source cluster, > the replication wouldn't happen. > Normally operator is supposed to retry the bulk load. But relying on human > retry is not robust solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17314) Limit total buffered size for all replication sources
[
https://issues.apache.org/jira/browse/HBASE-17314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754008#comment-15754008
]
Hadoop QA commented on HBASE-17314:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 51s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 2s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 41s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 99m 4s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 2m
0s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 148m 18s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.replication.multiwal.TestReplicationEndpointWithMultipleAsyncWAL |
| | hadoop.hbase.replication.multiwal.TestReplicationEndpointWithMultipleWAL |
| | hadoop.hbase.replication.TestReplicationEndpoint |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843563/HBASE-17314.v03.patch
|
| JIRA Issue | HBASE-17314 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux dc8fdffc23e6 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 6acbee1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4948/artifact/patchprocess/patch-unit-hbase-serv
[jira] [Commented] (HBASE-16398) optimize HRegion computeHDFSBlocksDistribution
[ https://issues.apache.org/jira/browse/HBASE-16398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754006#comment-15754006 ] Hudson commented on HBASE-16398: FAILURE: Integrated in Jenkins build HBase-1.4 #569 (See [https://builds.apache.org/job/HBase-1.4/569/]) HBASE-16398 optimize HRegion computeHDFSBlocksDistribution (binlijin: rev ed393964978969b6f1492a46ddae2893fd2875f4) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java > optimize HRegion computeHDFSBlocksDistribution > -- > > Key: HBASE-16398 > URL: https://issues.apache.org/jira/browse/HBASE-16398 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.0, 1.4.0 >Reporter: binlijin >Assignee: binlijin > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16398.branch-1.v1.patch, HBASE-16398.patch, > HBASE-16398_v2.patch, HBASE-16398_v3.patch, HBASE-16398_v4.patch, > HBASE-16398_v5.patch, LocatedBlockStatusComparison.java > > > First i assume there is no reference and link in a region family's directory. > Without the patch to computeHDFSBlocksDistribution for a region family, there > is 1+2*N rpc call, N is hfile numbers, The first rpc call is to > DistributedFileSystem#listStatus to get hfiles, for every hfile there is two > rpc call DistributedFileSystem#getFileStatus(path) and then > DistributedFileSystem#getFileBlockLocations(status, start, length). > With the patch to computeHDFSBlocksDistribution for a region family, there is > 2 rpc call, they are DistributedFileSystem#getFileStatus(path) and > DistributedFileSystem#listLocatedStatus(final Path p, final PathFilter > filter). > So if there is at least one hfile, with the patch, the rpc call will less. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17174) Refactor the AsyncProcess
[ https://issues.apache.org/jira/browse/HBASE-17174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-17174: -- Description: The following are reasons of reusing the pool and AP. # A update-heavy application, for example, loader, creates many BufferedMutator for batch updates. But these BufferedMutators can’t share a large threadpool because the shutdown() method will be called when closing any BufferedMutator. This patch adds a flag into BufferedMutatorParams for preventing calling the shutdown() method in BufferedMutatorImpl#close # The AsyncProcess has the powerful traffic control, but the control is against the single table currently. We should allow alternate traffic control implementation for advanced user who want more control. All suggestions are welcome. was: The following are reasons of reusing the pool and AP. # A update-heavy application, for example, loader, creates many BufferedMutator for batch updates. But these BufferedMutators can’t share a large threadpool because the shutdown() method will be called when closing any BufferedMutator. This patch adds a flag into BufferedMutatorParams for preventing calling the shutdown() method in BufferedMutatorImpl#close # The AsyncProcess has the powerful traffic control, but the control is against the single table currently. Because the AP is constructed at BufferedMutatorImpl's construction. This patch change the BufferedMutatorImpl's construction for reuse the AP so that the updates for different tables can be restricted by the same AP. Additionally, there are two changes(aren't included in latest patch) for #2. 1) The AP will be exposed to user. 2) A new method will be added to Connection for instantiating a AP. All suggestions are welcome. > Refactor the AsyncProcess > - > > Key: HBASE-17174 > URL: https://issues.apache.org/jira/browse/HBASE-17174 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17174.v0.patch, HBASE-17174.v1.patch, > HBASE-17174.v10.patch, HBASE-17174.v2.patch, HBASE-17174.v3.patch, > HBASE-17174.v4.patch, HBASE-17174.v5.patch, HBASE-17174.v6.patch, > HBASE-17174.v7.patch, HBASE-17174.v8.patch, HBASE-17174.v9.patch > > > The following are reasons of reusing the pool and AP. > # A update-heavy application, for example, loader, creates many > BufferedMutator for batch updates. But these BufferedMutators can’t share a > large threadpool because the shutdown() method will be called when closing > any BufferedMutator. This patch adds a flag into BufferedMutatorParams for > preventing calling the shutdown() method in BufferedMutatorImpl#close > # The AsyncProcess has the powerful traffic control, but the control is > against the single table currently. We should allow alternate traffic control > implementation for advanced user who want more control. > All suggestions are welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16859) Use Bytebuffer pool for non java clients specifically for scans/gets
[ https://issues.apache.org/jira/browse/HBASE-16859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753989#comment-15753989 ] ramkrishna.s.vasudevan commented on HBASE-16859: Ping!!. LAtest patch after review comments updated in the RB. > Use Bytebuffer pool for non java clients specifically for scans/gets > > > Key: HBASE-16859 > URL: https://issues.apache.org/jira/browse/HBASE-16859 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-16859_V1.patch, HBASE-16859_V2.patch, > HBASE-16859_V2.patch, HBASE-16859_V4.patch, HBASE-16859_V5.patch > > > In case of non java clients we still write the results and header into a on > demand byte[]. This can be changed to use the BBPool (onheap or offheap > buffer?). > But the basic problem is to identify if the response is for scans/gets. > - One easy way to do it is use the MethodDescriptor per Call and use the > name of the MethodDescriptor to identify it is a scan/get. But this will > pollute RpcServer by checking for scan/get type response. > - Other way is always set the result to cellScanner but we know that > isClientCellBlockSupported is going to false for non PB clients. So ignore > the cellscanner and go ahead with the results in PB. But this is not clean > - third one is that we already have a RpccallContext being passed to the RS. > In case of scan/gets/multiGets we already set a Rpccallback for shipped call. > So here on response we can check if the callback is not null and check for > isclientBlockSupported. In this case we can get the BB from the pool and > write the result and header to that BB. May be this looks clean? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17174) Refactor the AsyncProcess
[ https://issues.apache.org/jira/browse/HBASE-17174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-17174: -- Summary: Refactor the AsyncProcess (was: Use shared threadpool and AsyncProcess in BufferedMutatorImpl) > Refactor the AsyncProcess > - > > Key: HBASE-17174 > URL: https://issues.apache.org/jira/browse/HBASE-17174 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17174.v0.patch, HBASE-17174.v1.patch, > HBASE-17174.v10.patch, HBASE-17174.v2.patch, HBASE-17174.v3.patch, > HBASE-17174.v4.patch, HBASE-17174.v5.patch, HBASE-17174.v6.patch, > HBASE-17174.v7.patch, HBASE-17174.v8.patch, HBASE-17174.v9.patch > > > The following are reasons of reusing the pool and AP. > # A update-heavy application, for example, loader, creates many > BufferedMutator for batch updates. But these BufferedMutators can’t share a > large threadpool because the shutdown() method will be called when closing > any BufferedMutator. This patch adds a flag into BufferedMutatorParams for > preventing calling the shutdown() method in BufferedMutatorImpl#close > # The AsyncProcess has the powerful traffic control, but the control is > against the single table currently. Because the AP is constructed at > BufferedMutatorImpl's construction. This patch change the > BufferedMutatorImpl's construction for reuse the AP so that the updates for > different tables can be restricted by the same AP. > Additionally, there are two changes(aren't included in latest patch) for #2. > 1) The AP will be exposed to user. > 2) A new method will be added to Connection for instantiating a AP. > All suggestions are welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15248) BLOCKSIZE 4k should result in 4096 bytes on disk; i.e. fit inside a BucketCache 'block' of 4k
[ https://issues.apache.org/jira/browse/HBASE-15248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753974#comment-15753974 ] Anoop Sam John commented on HBASE-15248: BC is having diff sized buckets and slots in it. This sizes as per the possible HFile block sizes. By def we consider 4K, 8K... 512K.. But we give an additional +1KB size for each of the bucket size and so to a slot.. So a 4K is actually becoming 5K. And ya extra bytes are taken off by this cur header and next block header. So right now u have removed the eager fetch of next block header right? [~saint@gmail.com].. Still the cur block, header we will need? Am trying to solve issues under the parent so that I can make HBASE-17204 to happen. > BLOCKSIZE 4k should result in 4096 bytes on disk; i.e. fit inside a > BucketCache 'block' of 4k > - > > Key: HBASE-15248 > URL: https://issues.apache.org/jira/browse/HBASE-15248 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Reporter: stack > > Chatting w/ a gentleman named Daniel Pol who is messing w/ bucketcache, he > wants blocks to be the size specified in the configuration and no bigger. His > hardware set ups fetches pages of 4k and so a block that has 4k of payload > but has then a header and the header of the next block (which helps figure > whats next when scanning) ends up being 4203 bytes or something, and this > then then translates into two seeks per block fetch. > This issue is about what it would take to stay inside our configured size > boundary writing out blocks. > If not possible, give back better signal on what to do so you could fit > inside a particular constraint. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-15361) Remove unnecessary or Document constraints on BucketCache possible bucket sizes
[
https://issues.apache.org/jira/browse/HBASE-15361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John resolved HBASE-15361.
Resolution: Duplicate
> Remove unnecessary or Document constraints on BucketCache possible bucket
> sizes
>
>
> Key: HBASE-15361
> URL: https://issues.apache.org/jira/browse/HBASE-15361
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: deepankar
>Priority: Minor
>
> When we were trying to tune the bucket sizes
> {{hbase.bucketcache.bucket.sizes}} according to our workload, we encountered
> an issue due to the way offset is stored in the bucket entry. We divide the
> offset into integer base and byte value and it assumes that all bucket
> offsets will be a multiple of 256 (left shifting by 8). See the code below
> {code}
> long offset() { // Java has no unsigned numbers
> long o = ((long) offsetBase) & 0x;
> o += (((long) (offset1)) & 0xFF) << 32;
> return o << 8;
> }
> private void setOffset(long value) {
> assert (value & 0xFF) == 0;
> value >>= 8;
> offsetBase = (int) value;
> offset1 = (byte) (value >> 32);
> }
> {code}
> This was there to save 3 bytes per BucketEntry instead of using long and when
> there are no other fields in the Bucket Entry, but now there are lot of
> fields in the bucket entry , This not documented so we could either document
> the constraint that it should be a strict 256 bytes multiple of just go away
> with this constraint.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15361) Remove unnecessary or Document constraints on BucketCache possible bucket sizes
[
https://issues.apache.org/jira/browse/HBASE-15361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753961#comment-15753961
]
Anoop Sam John commented on HBASE-15361:
HBASE-16993 is trying to solve it.. Having offset state as long only.. Patch
avail there and hopefully can get committed soon.. Am resolving this issue as
a dup then
> Remove unnecessary or Document constraints on BucketCache possible bucket
> sizes
>
>
> Key: HBASE-15361
> URL: https://issues.apache.org/jira/browse/HBASE-15361
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
>Reporter: deepankar
>Priority: Minor
>
> When we were trying to tune the bucket sizes
> {{hbase.bucketcache.bucket.sizes}} according to our workload, we encountered
> an issue due to the way offset is stored in the bucket entry. We divide the
> offset into integer base and byte value and it assumes that all bucket
> offsets will be a multiple of 256 (left shifting by 8). See the code below
> {code}
> long offset() { // Java has no unsigned numbers
> long o = ((long) offsetBase) & 0x;
> o += (((long) (offset1)) & 0xFF) << 32;
> return o << 8;
> }
> private void setOffset(long value) {
> assert (value & 0xFF) == 0;
> value >>= 8;
> offsetBase = (int) value;
> offset1 = (byte) (value >> 32);
> }
> {code}
> This was there to save 3 bytes per BucketEntry instead of using long and when
> there are no other fields in the Bucket Entry, but now there are lot of
> fields in the bucket entry , This not documented so we could either document
> the constraint that it should be a strict 256 bytes multiple of just go away
> with this constraint.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17257) Add column-aliasing capability to hbase-client
[
https://issues.apache.org/jira/browse/HBASE-17257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753905#comment-15753905
]
Hadoop QA commented on HBASE-17257:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 46s
{color} | {color:red} hbase-client in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 51s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 46s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 59s {color}
| {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 105m 0s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 170m 39s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestBufferedMutator |
| | hadoop.hbase.client.TestAsyncGetMultiThread |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843554/HBASE-17257-v4.patch |
| JIRA Issue | HBASE-17257 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux fc5d0be3d913 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 6acbee1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build
[jira] [Commented] (HBASE-17324) PE Write workload results are wrong
[
https://issues.apache.org/jira/browse/HBASE-17324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753896#comment-15753896
]
Jingcheng Du commented on HBASE-17324:
--
ycsb has the same problem when testing latency. But it suggests to disable the
client buffer when testing latency.
I guess we can do the same thing in HBase? Enable the buffer when testing
throughput, and disable the buffer in latency.
> PE Write workload results are wrong
> ---
>
> Key: HBASE-17324
> URL: https://issues.apache.org/jira/browse/HBASE-17324
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>
> [~jmspaggi] found this issue of write latencies being unreasonable.
> The reason is, we are using BufferedMutator with size 2MB, so most writes
> return very quick because they get buffered. That's giving us avg. latency
> like 39us. Needs fixing.
> {noformat}
> 16/12/14 12:46:04 INFO hbase.PerformanceEvaluation: Latency (us) :
> mean=39,37, min=2,00, max=27193408,00, stdDev=3443,01, 50th=2,00, 75th=2,00,
> 95th=3,00, 99th=5,00, 99.9th=9157,00, 99.99th=39685,59, 99.999th=160751,28
> 16/12/14 12:46:04 INFO hbase.PerformanceEvaluation: Num measures (latency) :
> 1
> 16/12/14 12:46:04 INFO hbase.PerformanceEvaluation: Mean = 39,37
> Min = 2,00
> Max = 27193408,00
> StdDev= 3443,01
> 50th = 2,00
> 75th = 2,00
> 95th = 3,00
> 99th = 5,00
> 99.9th= 9157,00
> 99.99th = 39685,59
> 99.999th = 160751,28
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17282) Reduce the redundant requests to meta table
[
https://issues.apache.org/jira/browse/HBASE-17282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753884#comment-15753884
]
Hadoop QA commented on HBASE-17282:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 52s
{color} | {color:red} hbase-client in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 49s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 56s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 93m 57s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 137m 57s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12843559/HBASE-17282-v1.patch |
| JIRA Issue | HBASE-17282 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 22ee4319ac61 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 6acbee1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4945/artifact/patchprocess/branch-findbugs-hbase-client-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4945/testReport/ |
| modules | C: hbase-client hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HB
[jira] [Commented] (HBASE-17324) PE Write workload results are wrong
[
https://issues.apache.org/jira/browse/HBASE-17324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753864#comment-15753864
]
Jean-Marc Spaggiari commented on HBASE-17324:
-
Oh! I was not at all doing that for the credit Appy. I was just glad someone
considered this as important.
> PE Write workload results are wrong
> ---
>
> Key: HBASE-17324
> URL: https://issues.apache.org/jira/browse/HBASE-17324
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>
> [~jmspaggi] found this issue of write latencies being unreasonable.
> The reason is, we are using BufferedMutator with size 2MB, so most writes
> return very quick because they get buffered. That's giving us avg. latency
> like 39us. Needs fixing.
> {noformat}
> 16/12/14 12:46:04 INFO hbase.PerformanceEvaluation: Latency (us) :
> mean=39,37, min=2,00, max=27193408,00, stdDev=3443,01, 50th=2,00, 75th=2,00,
> 95th=3,00, 99th=5,00, 99.9th=9157,00, 99.99th=39685,59, 99.999th=160751,28
> 16/12/14 12:46:04 INFO hbase.PerformanceEvaluation: Num measures (latency) :
> 1
> 16/12/14 12:46:04 INFO hbase.PerformanceEvaluation: Mean = 39,37
> Min = 2,00
> Max = 27193408,00
> StdDev= 3443,01
> 50th = 2,00
> 75th = 2,00
> 95th = 3,00
> 99th = 5,00
> 99.9th= 9157,00
> 99.99th = 39685,59
> 99.999th = 160751,28
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16398) optimize HRegion computeHDFSBlocksDistribution
[ https://issues.apache.org/jira/browse/HBASE-16398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753793#comment-15753793 ] Hudson commented on HBASE-16398: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2140 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2140/]) HBASE-16398 optimize HRegion computeHDFSBlocksDistribution (binlijin: rev 6acbee179feca63d12ea3fff6e9f25a5c044c467) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java > optimize HRegion computeHDFSBlocksDistribution > -- > > Key: HBASE-16398 > URL: https://issues.apache.org/jira/browse/HBASE-16398 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.0, 1.4.0 >Reporter: binlijin >Assignee: binlijin > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16398.branch-1.v1.patch, HBASE-16398.patch, > HBASE-16398_v2.patch, HBASE-16398_v3.patch, HBASE-16398_v4.patch, > HBASE-16398_v5.patch, LocatedBlockStatusComparison.java > > > First i assume there is no reference and link in a region family's directory. > Without the patch to computeHDFSBlocksDistribution for a region family, there > is 1+2*N rpc call, N is hfile numbers, The first rpc call is to > DistributedFileSystem#listStatus to get hfiles, for every hfile there is two > rpc call DistributedFileSystem#getFileStatus(path) and then > DistributedFileSystem#getFileBlockLocations(status, start, length). > With the patch to computeHDFSBlocksDistribution for a region family, there is > 2 rpc call, they are DistributedFileSystem#getFileStatus(path) and > DistributedFileSystem#listLocatedStatus(final Path p, final PathFilter > filter). > So if there is at least one hfile, with the patch, the rpc call will less. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
