[jira] [Commented] (HBASE-12444) Total number of requests overflow because it's int
[ https://issues.apache.org/jira/browse/HBASE-12444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281416#comment-16281416 ] Anoop Sam John commented on HBASE-12444: LGTM > Total number of requests overflow because it's int > -- > > Key: HBASE-12444 > URL: https://issues.apache.org/jira/browse/HBASE-12444 > Project: HBase > Issue Type: Bug > Components: hbck, master, regionserver >Reporter: yunjiong zhao >Assignee: Chia-Ping Tsai >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-12444.v0.patch, hbase-12444-v1.patch, > hbase-12444.patch > > > When running hbck, I noticed "Number of requests" was wrong: > Average load: 466.41237113402065 > Number of requests: -1835941345 > Number of regions: 45242 > Number of regions in transition: 0 > The root cause is it use int, and clearly it overflowed. > I'll update a patch later. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19357) Bucket cache no longer L2 for LRU cache
[ https://issues.apache.org/jira/browse/HBASE-19357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281405#comment-16281405 ] Anoop Sam John commented on HBASE-19357: Will do the on heap LRU (old L1) vs the file mode bucket cache.. With this patch and with out. To see the impact of keeping the META table data in file cache not in LRU on heap.. Ya the OS cache will be a key there. In initial tests there may be like the OS cache is big enough to hold the entire data in bucket cache (The data size wont be more than 25GB).. Once I have numbers abt that, will see possible way to limit the OS cache availability (as in real cluster cases) and see the numbers. Tks all.. Will commit then as Stack said with a TODO abt the this test and possible come back here as needed. > Bucket cache no longer L2 for LRU cache > --- > > Key: HBASE-19357 > URL: https://issues.apache.org/jira/browse/HBASE-19357 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19357.patch, HBASE-19357.patch, > HBASE-19357_V2.patch, HBASE-19357_V3.patch, HBASE-19357_V3.patch > > > When Bucket cache is used, by default we dont configure it as an L2 cache > alone. The default setting is combined mode ON where the data blocks to > Bucket cache and index/bloom blocks go to LRU cache. But there is a way to > turn this off and make LRU as L1 and Bucket cache as a victim handler for L1. > It will be just L2. > After the off heap read path optimization Bucket cache is no longer slower > compared to L1. We have test results on data sizes from 12 GB. The Alibaba > use case was also with 12 GB and they have observed a ~30% QPS improve over > the LRU cache. > This issue is to remove the option for combined mode = false. So when Bucket > cache is in use, data blocks will go to it only and LRU will get only index > /meta/bloom blocks. Bucket cache will no longer be configured as a victim > handler for LRU. > Note : WHen external cache is in use, there only the L1 L2 thing comes. LRU > will be L1 and external cache act as its L2. That make full sense. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19423) Replication entries are not filtered correctly when replication scope is set through WAL Co-processor
[
https://issues.apache.org/jira/browse/HBASE-19423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281401#comment-16281401
]
Anoop Sam John commented on HBASE-19423:
My concern was on the general approach.. What the test is doing in the CP hook.
{quote}
I have a scenario where in a user table some column families are system
generated and these families data need not to be replicated.
Because these are system generated column families it would not be appropriate
to expect user to disable replication for these families.
{quote}
Well the replication enable is at CF level. One can set on HCD. By default
the replication is disabled for any CF. ie REPLICATION_SCOPE_LOCAL
One can enable this by setting REPLICATION_SCOPE_GLOBAL on those CFs which has
to be replicated. So in ur case when the table is having some of the system
generated CFs the replication is OFF on them by default.. And this is what u
need. The other CFs on the user table, which the user created, the use can opt
to set replication scope on those only which he wants.
So in general I dont see any need NOT to set the replication global scope
enabled on any of the CFs of the table and doing the selective replication of
some of the CF data using the CP hook. That is very very strange usage for me.
Hope am making it clear. Ya seems in master branch, the place where the scope
been set on the WALKey seems different so there is no reset happening. But
setting the scope on the walkey, I really doubt whether that can be a work of
the CPs. This is some thing the core code has to do. Even on master, the
WALKey is still marked @InterfaceAudience.Private. Means we really dont expect
CPs to use all methods in that. This is an issue that we over expose things to
CPs. I agree.. So pls consider ur solution 1st I would say.
> Replication entries are not filtered correctly when replication scope is set
> through WAL Co-processor
> -
>
> Key: HBASE-19423
> URL: https://issues.apache.org/jira/browse/HBASE-19423
> Project: HBase
> Issue Type: Bug
>Reporter: Mohammad Arshad
> Labels: Replication, WAL
> Fix For: 1.4.0, 1.3.2
>
> Attachments: HBASE-19423-branch-1.3-001.patch,
> HBASE-19423-branch-1.4-001.patch, HBASE-19423-master-001-test.patch
>
>
> Replicaion scope set in WALObserver is getting reset in
> Replication.scopeWALEdits().
> Because of this problem custom implementation of WALObserver can not be used
> as a replication filter.
> Suppose WALObserver implementation has logic to filter all entries from
> family f2
> {code}
> // Filter all family f2 rows
> public static class ReplicationFilterWALCoprocessor extends BaseWALObserver
> {
> @Override
> public boolean preWALWrite(ObserverContext WALCoprocessorEnvironment> ctx,
> HRegionInfo info, WALKey logKey, WALEdit logEdit) throws IOException {
> ArrayList cells = logEdit.getCells();
> for (Cell cell : cells) {
> byte[] fam = CellUtil.cloneFamily(cell);
> if ("f2".equals(Bytes.toString(fam))) {
> NavigableMap scopes = logKey.getScopes();
> if (scopes == null) {
> logKey.setScopes(new TreeMap Integer>(Bytes.BYTES_COMPARATOR));
> }
> logKey.getScopes().put(fam, HConstants.REPLICATION_SCOPE_LOCAL);
> }
> }
> return false;
> }
> }
> {code}
> This logic can not work as
> {{org.apache.hadoop.hbase.replication.regionserver.Replication.scopeWALEdits()}}
> recreates and populates scopes.
> *SOLUTION:*
> In Replication.scopeWALEdits(), create scopes map only if WALKey does not
> have it.
> {code}
> NavigableMap scopes = logKey.getScopes();
> if (scopes == null) {
> scopes = new TreeMap(Bytes.BYTES_COMPARATOR);
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19134) Make WALKey an Interface; expose Read-Only version to CPs
[ https://issues.apache.org/jira/browse/HBASE-19134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19134: -- Attachment: HBASE-19134.master.001.patch > Make WALKey an Interface; expose Read-Only version to CPs > - > > Key: HBASE-19134 > URL: https://issues.apache.org/jira/browse/HBASE-19134 > Project: HBase > Issue Type: Bug > Components: Coprocessors, wal >Reporter: stack >Assignee: stack > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19134.master.001.patch > > > WALKey has been made IA.Private in hbase2. Even so, given we don't have an > alternative to expose at this time, it is exposed to coprocessors still at a > few (now deprecated) locations. > In review of HBASE-18770, [~chia7712] makes reasonable suggestion that what > we expose to CPs be a read-only WALKey. He gets pushback on doing this for > hbase2 (Do we even want to expose WALKey to CPs, is WALKey right going > forward, etc.). Chia-Ping comes back w/ the below (copied from HBASE-18770): > What we want to fix for WALKey are shown below. > * expose some methods to CP user safety > * refactor/redo > As I see it, adding an interface exposed to CP user for WALKey is a right > choice because it can bring some benefit. > * We can expose part of WALKey's methods to CP users - a read-only interface > or an interface with some modifiable setting. > * The related CP hooks won't be deprecated > * Doing the refactor for WALKey doesn't essentially impact the CP hook after > 2.0 release. > Although, it will be better to redo WALKey before 2.0 release. In short, I > think it warrants such an interface. > (We both agree this would be a load of work given WALKey is written to > HFiles. Warrants a look though). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19134) Make WALKey an Interface; expose Read-Only version to CPs
[ https://issues.apache.org/jira/browse/HBASE-19134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281397#comment-16281397 ] stack commented on HBASE-19134: --- .001 Not for review. Seeing what is broke. This might work after all. > Make WALKey an Interface; expose Read-Only version to CPs > - > > Key: HBASE-19134 > URL: https://issues.apache.org/jira/browse/HBASE-19134 > Project: HBase > Issue Type: Bug > Components: Coprocessors, wal >Reporter: stack >Assignee: stack > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19134.master.001.patch > > > WALKey has been made IA.Private in hbase2. Even so, given we don't have an > alternative to expose at this time, it is exposed to coprocessors still at a > few (now deprecated) locations. > In review of HBASE-18770, [~chia7712] makes reasonable suggestion that what > we expose to CPs be a read-only WALKey. He gets pushback on doing this for > hbase2 (Do we even want to expose WALKey to CPs, is WALKey right going > forward, etc.). Chia-Ping comes back w/ the below (copied from HBASE-18770): > What we want to fix for WALKey are shown below. > * expose some methods to CP user safety > * refactor/redo > As I see it, adding an interface exposed to CP user for WALKey is a right > choice because it can bring some benefit. > * We can expose part of WALKey's methods to CP users - a read-only interface > or an interface with some modifiable setting. > * The related CP hooks won't be deprecated > * Doing the refactor for WALKey doesn't essentially impact the CP hook after > 2.0 release. > Although, it will be better to redo WALKey before 2.0 release. In short, I > think it warrants such an interface. > (We both agree this would be a load of work given WALKey is written to > HFiles. Warrants a look though). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-12444) Total number of requests overflow because it's int
[ https://issues.apache.org/jira/browse/HBASE-12444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-12444: --- Hadoop Flags: (was: Incompatible change) > Total number of requests overflow because it's int > -- > > Key: HBASE-12444 > URL: https://issues.apache.org/jira/browse/HBASE-12444 > Project: HBase > Issue Type: Bug > Components: hbck, master, regionserver >Reporter: yunjiong zhao >Assignee: Chia-Ping Tsai >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-12444.v0.patch, hbase-12444-v1.patch, > hbase-12444.patch > > > When running hbck, I noticed "Number of requests" was wrong: > Average load: 466.41237113402065 > Number of requests: -1835941345 > Number of regions: 45242 > Number of regions in transition: 0 > The root cause is it use int, and clearly it overflowed. > I'll update a patch later. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-12444) Total number of requests overflow because it's int
[ https://issues.apache.org/jira/browse/HBASE-12444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-12444: --- Status: Patch Available (was: Open) > Total number of requests overflow because it's int > -- > > Key: HBASE-12444 > URL: https://issues.apache.org/jira/browse/HBASE-12444 > Project: HBase > Issue Type: Bug > Components: hbck, master, regionserver >Reporter: yunjiong zhao >Assignee: Chia-Ping Tsai >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-12444.v0.patch, hbase-12444-v1.patch, > hbase-12444.patch > > > When running hbck, I noticed "Number of requests" was wrong: > Average load: 466.41237113402065 > Number of requests: -1835941345 > Number of regions: 45242 > Number of regions in transition: 0 > The root cause is it use int, and clearly it overflowed. > I'll update a patch later. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-12444) Total number of requests overflow because it's int
[ https://issues.apache.org/jira/browse/HBASE-12444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-12444: --- Attachment: HBASE-12444.v0.patch previous name: getRequest*s*Count new name: getRequestCount > Total number of requests overflow because it's int > -- > > Key: HBASE-12444 > URL: https://issues.apache.org/jira/browse/HBASE-12444 > Project: HBase > Issue Type: Bug > Components: hbck, master, regionserver >Reporter: yunjiong zhao >Assignee: Chia-Ping Tsai >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-12444.v0.patch, hbase-12444-v1.patch, > hbase-12444.patch > > > When running hbck, I noticed "Number of requests" was wrong: > Average load: 466.41237113402065 > Number of requests: -1835941345 > Number of regions: 45242 > Number of regions in transition: 0 > The root cause is it use int, and clearly it overflowed. > I'll update a patch later. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19357) Bucket cache no longer L2 for LRU cache
[ https://issues.apache.org/jira/browse/HBASE-19357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281393#comment-16281393 ] ramkrishna.s.vasudevan commented on HBASE-19357: Just to re emphasis we have done multiple tests with less data and data ~200G and all data in bucket cache. Offheap in all cases was atleast 12% better. We have also done long running tests for more than 24 hours across 3 nodes with 16G bucket cache size and 200G data. There were no issues in the log or any data corruption. File mode yes - if all from OS cache will be similar to L1 unless some meta blocks are evicted and has to be read back. Thanks. > Bucket cache no longer L2 for LRU cache > --- > > Key: HBASE-19357 > URL: https://issues.apache.org/jira/browse/HBASE-19357 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19357.patch, HBASE-19357.patch, > HBASE-19357_V2.patch, HBASE-19357_V3.patch, HBASE-19357_V3.patch > > > When Bucket cache is used, by default we dont configure it as an L2 cache > alone. The default setting is combined mode ON where the data blocks to > Bucket cache and index/bloom blocks go to LRU cache. But there is a way to > turn this off and make LRU as L1 and Bucket cache as a victim handler for L1. > It will be just L2. > After the off heap read path optimization Bucket cache is no longer slower > compared to L1. We have test results on data sizes from 12 GB. The Alibaba > use case was also with 12 GB and they have observed a ~30% QPS improve over > the LRU cache. > This issue is to remove the option for combined mode = false. So when Bucket > cache is in use, data blocks will go to it only and LRU will get only index > /meta/bloom blocks. Bucket cache will no longer be configured as a victim > handler for LRU. > Note : WHen external cache is in use, there only the L1 L2 thing comes. LRU > will be L1 and external cache act as its L2. That make full sense. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19449) Implement SLF4J and SLF4J Parameter Substitution
[
https://issues.apache.org/jira/browse/HBASE-19449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281392#comment-16281392
]
Hadoop QA commented on HBASE-19449:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
25s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
15s{color} | {color:red} hbase-server: The patch generated 1 new + 14 unchanged
- 5 fixed = 15 total (was 19) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
47s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
51m 29s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 2.7.4 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 93m
50s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}167m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19449 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12901002/HBASE-19449.1.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 601609bb7894 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 97d17ae195 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/10281/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/10281/artifact/patchprocess/whitespace-eol.txt
[jira] [Assigned] (HBASE-12444) Total number of requests overflow because it's int
[ https://issues.apache.org/jira/browse/HBASE-12444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai reassigned HBASE-12444: -- Assignee: Chia-Ping Tsai (was: yunjiong zhao) > Total number of requests overflow because it's int > -- > > Key: HBASE-12444 > URL: https://issues.apache.org/jira/browse/HBASE-12444 > Project: HBase > Issue Type: Bug > Components: hbck, master, regionserver >Reporter: yunjiong zhao >Assignee: Chia-Ping Tsai >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: hbase-12444-v1.patch, hbase-12444.patch > > > When running hbck, I noticed "Number of requests" was wrong: > Average load: 466.41237113402065 > Number of requests: -1835941345 > Number of regions: 45242 > Number of regions in transition: 0 > The root cause is it use int, and clearly it overflowed. > I'll update a patch later. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-12444) Total number of requests overflow because it's int
[ https://issues.apache.org/jira/browse/HBASE-12444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-12444: --- Fix Version/s: 2.0.0-beta-1 > Total number of requests overflow because it's int > -- > > Key: HBASE-12444 > URL: https://issues.apache.org/jira/browse/HBASE-12444 > Project: HBase > Issue Type: Bug > Components: hbck, master, regionserver >Reporter: yunjiong zhao >Assignee: Chia-Ping Tsai >Priority: Minor > Fix For: 2.0.0-beta-1 > > Attachments: hbase-12444-v1.patch, hbase-12444.patch > > > When running hbck, I noticed "Number of requests" was wrong: > Average load: 466.41237113402065 > Number of requests: -1835941345 > Number of regions: 45242 > Number of regions in transition: 0 > The root cause is it use int, and clearly it overflowed. > I'll update a patch later. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-12444) Total number of requests overflow because it's int
[
https://issues.apache.org/jira/browse/HBASE-12444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281391#comment-16281391
]
Chia-Ping Tsai commented on HBASE-12444:
The field in the protobuf had be changed from uint32 to uint64 (HBASE-13661).
The remaining works is to deprecate the shorter count and add a longer one for
{{ClusterStatus}}. Let me pick this up.
> Total number of requests overflow because it's int
> --
>
> Key: HBASE-12444
> URL: https://issues.apache.org/jira/browse/HBASE-12444
> Project: HBase
> Issue Type: Bug
> Components: hbck, master, regionserver
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
>Priority: Minor
> Fix For: 2.0.0-beta-1
>
> Attachments: hbase-12444-v1.patch, hbase-12444.patch
>
>
> When running hbck, I noticed "Number of requests" was wrong:
> Average load: 466.41237113402065
> Number of requests: -1835941345
> Number of regions: 45242
> Number of regions in transition: 0
> The root cause is it use int, and clearly it overflowed.
> I'll update a patch later.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19340) Backport missing options in shell
[ https://issues.apache.org/jira/browse/HBASE-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281387#comment-16281387 ] Hudson commented on HBASE-19340: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #313 (See [https://builds.apache.org/job/HBase-1.3-IT/313/]) HBASE-19340 Backport missing options in shell (chia7712: rev 2ff895c10aa87baa8604a4b13eec028b18f7bef1) * (edit) hbase-shell/src/test/ruby/hbase/admin_test.rb * (edit) hbase-shell/src/main/ruby/hbase/admin.rb > Backport missing options in shell > - > > Key: HBASE-19340 > URL: https://issues.apache.org/jira/browse/HBASE-19340 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 1.2.6 >Reporter: zhaoyuan >Assignee: zhaoyuan > Fix For: 1.3.2, 1.2.7 > > Attachments: HBASE-19340-branch-1.2.batch, > HBASE-19340-branch-1.2.batch, HBASE-19340-branch-1.3.batch > > > Recently I wanna try to alter the split policy for a table on my cluster > which version is 1.2.6 and as far as I know The SPLIT_POLICY is an attribute > of the HTable so I run the command in hbase shell console below. > alter 'tablex',SPLIT_POLICY => > 'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy' > However, It gave the information like this and I confused > Unknown argument ignored: SPLIT_POLICY > Updating all regions with the new schema... > So I check the source code That admin.rb might miss the setting for this > argument . > htd.setMaxFileSize(JLong.valueOf(arg.delete(MAX_FILESIZE))) if > arg[MAX_FILESIZE] > htd.setReadOnly(JBoolean.valueOf(arg.delete(READONLY))) if arg[READONLY] > ... > So I think it may be a bug ,is it? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19301) Provide way for CPs to create short circuited connection with custom configurations
[ https://issues.apache.org/jira/browse/HBASE-19301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281385#comment-16281385 ] Anoop Sam John commented on HBASE-19301: Nice warns and clear messages to user.. +1 Stack. > Provide way for CPs to create short circuited connection with custom > configurations > --- > > Key: HBASE-19301 > URL: https://issues.apache.org/jira/browse/HBASE-19301 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19301-addendum.patch, > HBASE-19301.master.001.patch, HBASE-19301.patch, HBASE-19301_V2.patch, > HBASE-19301_V2.patch > > > Over in HBASE-18359 we have discussions for this. > Right now HBase provide getConnection() in RegionCPEnv, MasterCPEnv etc. But > this returns a pre created connection (per server). This uses the configs at > hbase-site.xml at that server. > Phoenix needs creating connection in CP with some custom configs. Having this > custom changes in hbase-site.xml is harmful as that will affect all > connections been created at that server. > This issue is for providing an overloaded getConnection(Configuration) API -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19340) Backport missing options in shell
[ https://issues.apache.org/jira/browse/HBASE-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281384#comment-16281384 ] Hudson commented on HBASE-19340: FAILURE: Integrated in Jenkins build HBase-1.2-IT #1037 (See [https://builds.apache.org/job/HBase-1.2-IT/1037/]) HBASE-19340 Backport missing options in shell (chia7712: rev 45e230959bf141550cadd69a4c0a078c6fd1c2ee) * (edit) hbase-shell/src/test/ruby/hbase/admin_test.rb * (edit) hbase-shell/src/main/ruby/hbase/admin.rb > Backport missing options in shell > - > > Key: HBASE-19340 > URL: https://issues.apache.org/jira/browse/HBASE-19340 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 1.2.6 >Reporter: zhaoyuan >Assignee: zhaoyuan > Fix For: 1.3.2, 1.2.7 > > Attachments: HBASE-19340-branch-1.2.batch, > HBASE-19340-branch-1.2.batch, HBASE-19340-branch-1.3.batch > > > Recently I wanna try to alter the split policy for a table on my cluster > which version is 1.2.6 and as far as I know The SPLIT_POLICY is an attribute > of the HTable so I run the command in hbase shell console below. > alter 'tablex',SPLIT_POLICY => > 'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy' > However, It gave the information like this and I confused > Unknown argument ignored: SPLIT_POLICY > Updating all regions with the new schema... > So I check the source code That admin.rb might miss the setting for this > argument . > htd.setMaxFileSize(JLong.valueOf(arg.delete(MAX_FILESIZE))) if > arg[MAX_FILESIZE] > htd.setReadOnly(JBoolean.valueOf(arg.delete(READONLY))) if arg[READONLY] > ... > So I think it may be a bug ,is it? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19340) Backport missing options in shell
[ https://issues.apache.org/jira/browse/HBASE-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaoyuan updated HBASE-19340: - Release Note: Add missing constants configuration in shell thru backport. (was: Add split policy and flush policy configuration in shell) > Backport missing options in shell > - > > Key: HBASE-19340 > URL: https://issues.apache.org/jira/browse/HBASE-19340 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 1.2.6 >Reporter: zhaoyuan >Assignee: zhaoyuan > Fix For: 1.3.2, 1.2.7 > > Attachments: HBASE-19340-branch-1.2.batch, > HBASE-19340-branch-1.2.batch, HBASE-19340-branch-1.3.batch > > > Recently I wanna try to alter the split policy for a table on my cluster > which version is 1.2.6 and as far as I know The SPLIT_POLICY is an attribute > of the HTable so I run the command in hbase shell console below. > alter 'tablex',SPLIT_POLICY => > 'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy' > However, It gave the information like this and I confused > Unknown argument ignored: SPLIT_POLICY > Updating all regions with the new schema... > So I check the source code That admin.rb might miss the setting for this > argument . > htd.setMaxFileSize(JLong.valueOf(arg.delete(MAX_FILESIZE))) if > arg[MAX_FILESIZE] > htd.setReadOnly(JBoolean.valueOf(arg.delete(READONLY))) if arg[READONLY] > ... > So I think it may be a bug ,is it? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19443) HBase Client gets stuck - Cannot get replica 0 location for...
[
https://issues.apache.org/jira/browse/HBASE-19443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lakshmiprasad D updated HBASE-19443:
Priority: Critical (was: Major)
> HBase Client gets stuck - Cannot get replica 0 location for...
> --
>
> Key: HBASE-19443
> URL: https://issues.apache.org/jira/browse/HBASE-19443
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.4
>Reporter: Lakshmiprasad D
>Priority: Critical
>
> When the loader is loading data to HBase, without any forewarning that client
> gets stuck and prints the below message continuously:
> [ERROR] AsyncProcess[hconnection-0x71f5455a-shared--pool20-t40846] Cannot get
> replica 0 location for
> {"totalColumns":20,"families":{"v":[{"timestamp":151258499,"tag":[],"qualifier":"0","vlen":4},{"timestamp":151258499,"tag":[],"qualifier":"uid","vlen":8},{"timestamp":151258499,"tag":[],"qualifier":"1","vlen":8},{"timestamp":151258499,"tag":[],"qualifier":"2","vlen":11}]},"row":"C\\x99L[5\\x80\\x00\\x00"}
> At this time no other loaders are impacted. Restart of the loader seems to
> solve the issue until next occurence.
> Upon doing jstack at this time on the loader, it is observed that process is
> stuck on a object monitor inside 'AsyncProcess.java' class.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19340) Backport missing options in shell
[ https://issues.apache.org/jira/browse/HBASE-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-19340: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Push to branch-1.2 and 1.3. [~smartZY] Would you please update the release note? Thanks for your contribution. > Backport missing options in shell > - > > Key: HBASE-19340 > URL: https://issues.apache.org/jira/browse/HBASE-19340 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 1.2.6 >Reporter: zhaoyuan >Assignee: zhaoyuan > Fix For: 1.3.2, 1.2.7 > > Attachments: HBASE-19340-branch-1.2.batch, > HBASE-19340-branch-1.2.batch, HBASE-19340-branch-1.3.batch > > > Recently I wanna try to alter the split policy for a table on my cluster > which version is 1.2.6 and as far as I know The SPLIT_POLICY is an attribute > of the HTable so I run the command in hbase shell console below. > alter 'tablex',SPLIT_POLICY => > 'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy' > However, It gave the information like this and I confused > Unknown argument ignored: SPLIT_POLICY > Updating all regions with the new schema... > So I check the source code That admin.rb might miss the setting for this > argument . > htd.setMaxFileSize(JLong.valueOf(arg.delete(MAX_FILESIZE))) if > arg[MAX_FILESIZE] > htd.setReadOnly(JBoolean.valueOf(arg.delete(READONLY))) if arg[READONLY] > ... > So I think it may be a bug ,is it? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19340) Backport missing options in shell
[ https://issues.apache.org/jira/browse/HBASE-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-19340: --- Summary: Backport missing options in shell (was: SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell) > Backport missing options in shell > - > > Key: HBASE-19340 > URL: https://issues.apache.org/jira/browse/HBASE-19340 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 1.2.6 >Reporter: zhaoyuan >Assignee: zhaoyuan > Fix For: 1.3.2, 1.2.7 > > Attachments: HBASE-19340-branch-1.2.batch, > HBASE-19340-branch-1.2.batch, HBASE-19340-branch-1.3.batch > > > Recently I wanna try to alter the split policy for a table on my cluster > which version is 1.2.6 and as far as I know The SPLIT_POLICY is an attribute > of the HTable so I run the command in hbase shell console below. > alter 'tablex',SPLIT_POLICY => > 'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy' > However, It gave the information like this and I confused > Unknown argument ignored: SPLIT_POLICY > Updating all regions with the new schema... > So I check the source code That admin.rb might miss the setting for this > argument . > htd.setMaxFileSize(JLong.valueOf(arg.delete(MAX_FILESIZE))) if > arg[MAX_FILESIZE] > htd.setReadOnly(JBoolean.valueOf(arg.delete(READONLY))) if arg[READONLY] > ... > So I think it may be a bug ,is it? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19447) INFO level logging of GetClusterStatus from HMaster is too chatty
[ https://issues.apache.org/jira/browse/HBASE-19447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281375#comment-16281375 ] Hudson commented on HBASE-19447: SUCCESS: Integrated in Jenkins build HBase-1.4 #1055 (See [https://builds.apache.org/job/HBase-1.4/1055/]) HBASE-19447 INFO level logging of GetClusterStatus from HMaster is too (apurtell: rev 099de9088b5bcee3d2ce1c96f940e2f198c775c0) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java > INFO level logging of GetClusterStatus from HMaster is too chatty > - > > Key: HBASE-19447 > URL: https://issues.apache.org/jira/browse/HBASE-19447 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0 > > > Should be DEBUG. Should be wrapped with a logLevel check. > ./hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java:2409: > LOG.info(getClientIdAuditPrefix() + " get ClusterStatus, status=" + > status); -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19446) Misspelled 'default' in SimpleRpcScheduler
[
https://issues.apache.org/jira/browse/HBASE-19446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281376#comment-16281376
]
Hudson commented on HBASE-19446:
SUCCESS: Integrated in Jenkins build HBase-1.4 #1055 (See
[https://builds.apache.org/job/HBase-1.4/1055/])
HBASE-19446 Misspelled 'default' in SimpleRpcScheduler (apurtell: rev
8731d9e3e8f16cb3487336d0e96a02ed4ee5834f)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestSimpleRpcScheduler.java
> Misspelled 'default' in SimpleRpcScheduler
> --
>
> Key: HBASE-19446
> URL: https://issues.apache.org/jira/browse/HBASE-19446
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Trivial
> Fix For: 1.4.0
>
>
> Found "deafult' in logs, should be "default" of course. Looks bad, going to
> fix.
> $ find . -name '*.java' -exec fgrep -Hn deafult {} \;
> ./hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java:85:
> callExecutor = new RWQueueRpcExecutor("deafult.RWQ", Math.max(2,
> handlerCount),
> ./hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java:89:
> callExecutor = new FastPathBalancedQueueRpcExecutor("deafult.FPBQ",
> handlerCount,
> ./hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java:92:
> callExecutor = new BalancedQueueRpcExecutor("deafult.BQ",
> handlerCount, maxQueueLength,
> ./hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestSimpleRpcScheduler.java:413:
> envEdge.threadNamePrefixs.add("RpcServer.deafult.FPBQ.Codel.handler");
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19340) SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell
[ https://issues.apache.org/jira/browse/HBASE-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-19340: --- Fix Version/s: (was: 1.2.8) 1.2.7 > SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell > --- > > Key: HBASE-19340 > URL: https://issues.apache.org/jira/browse/HBASE-19340 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 1.2.6 >Reporter: zhaoyuan >Assignee: zhaoyuan > Fix For: 1.3.2, 1.2.7 > > Attachments: HBASE-19340-branch-1.2.batch, > HBASE-19340-branch-1.2.batch, HBASE-19340-branch-1.3.batch > > > Recently I wanna try to alter the split policy for a table on my cluster > which version is 1.2.6 and as far as I know The SPLIT_POLICY is an attribute > of the HTable so I run the command in hbase shell console below. > alter 'tablex',SPLIT_POLICY => > 'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy' > However, It gave the information like this and I confused > Unknown argument ignored: SPLIT_POLICY > Updating all regions with the new schema... > So I check the source code That admin.rb might miss the setting for this > argument . > htd.setMaxFileSize(JLong.valueOf(arg.delete(MAX_FILESIZE))) if > arg[MAX_FILESIZE] > htd.setReadOnly(JBoolean.valueOf(arg.delete(READONLY))) if arg[READONLY] > ... > So I think it may be a bug ,is it? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19440) Not able to enable balancer with RSGroups once disabled
[ https://issues.apache.org/jira/browse/HBASE-19440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281373#comment-16281373 ] Abhishek Singh Chouhan commented on HBASE-19440: Thanks [~apurtell] [~tedyu] :) > Not able to enable balancer with RSGroups once disabled > --- > > Key: HBASE-19440 > URL: https://issues.apache.org/jira/browse/HBASE-19440 > Project: HBase > Issue Type: Bug >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Fix For: 1.4.0 > > Attachments: HBASE-19440.branch-1.001.patch > > > Once the balancer is disabled, trying to switch it back on doesn't work since > the prebalanceswitch coprocessor hook is incorrectly always returning false. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19340) SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell
[ https://issues.apache.org/jira/browse/HBASE-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281374#comment-16281374 ] Chia-Ping Tsai commented on HBASE-19340: LGTM > SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell > --- > > Key: HBASE-19340 > URL: https://issues.apache.org/jira/browse/HBASE-19340 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 1.2.6 >Reporter: zhaoyuan >Assignee: zhaoyuan > Fix For: 1.3.2, 1.2.8 > > Attachments: HBASE-19340-branch-1.2.batch, > HBASE-19340-branch-1.2.batch, HBASE-19340-branch-1.3.batch > > > Recently I wanna try to alter the split policy for a table on my cluster > which version is 1.2.6 and as far as I know The SPLIT_POLICY is an attribute > of the HTable so I run the command in hbase shell console below. > alter 'tablex',SPLIT_POLICY => > 'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy' > However, It gave the information like this and I confused > Unknown argument ignored: SPLIT_POLICY > Updating all regions with the new schema... > So I check the source code That admin.rb might miss the setting for this > argument . > htd.setMaxFileSize(JLong.valueOf(arg.delete(MAX_FILESIZE))) if > arg[MAX_FILESIZE] > htd.setReadOnly(JBoolean.valueOf(arg.delete(READONLY))) if arg[READONLY] > ... > So I think it may be a bug ,is it? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19340) SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell
[ https://issues.apache.org/jira/browse/HBASE-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-19340: --- Component/s: shell > SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell > --- > > Key: HBASE-19340 > URL: https://issues.apache.org/jira/browse/HBASE-19340 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 1.2.6 >Reporter: zhaoyuan >Assignee: zhaoyuan > Fix For: 1.3.2, 1.2.8 > > Attachments: HBASE-19340-branch-1.2.batch, > HBASE-19340-branch-1.2.batch, HBASE-19340-branch-1.3.batch > > > Recently I wanna try to alter the split policy for a table on my cluster > which version is 1.2.6 and as far as I know The SPLIT_POLICY is an attribute > of the HTable so I run the command in hbase shell console below. > alter 'tablex',SPLIT_POLICY => > 'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy' > However, It gave the information like this and I confused > Unknown argument ignored: SPLIT_POLICY > Updating all regions with the new schema... > So I check the source code That admin.rb might miss the setting for this > argument . > htd.setMaxFileSize(JLong.valueOf(arg.delete(MAX_FILESIZE))) if > arg[MAX_FILESIZE] > htd.setReadOnly(JBoolean.valueOf(arg.delete(READONLY))) if arg[READONLY] > ... > So I think it may be a bug ,is it? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19340) SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell
[ https://issues.apache.org/jira/browse/HBASE-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-19340: --- Fix Version/s: 1.3.2 > SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell > --- > > Key: HBASE-19340 > URL: https://issues.apache.org/jira/browse/HBASE-19340 > Project: HBase > Issue Type: Bug > Components: shell >Affects Versions: 1.2.6 >Reporter: zhaoyuan >Assignee: zhaoyuan > Fix For: 1.3.2, 1.2.8 > > Attachments: HBASE-19340-branch-1.2.batch, > HBASE-19340-branch-1.2.batch, HBASE-19340-branch-1.3.batch > > > Recently I wanna try to alter the split policy for a table on my cluster > which version is 1.2.6 and as far as I know The SPLIT_POLICY is an attribute > of the HTable so I run the command in hbase shell console below. > alter 'tablex',SPLIT_POLICY => > 'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy' > However, It gave the information like this and I confused > Unknown argument ignored: SPLIT_POLICY > Updating all regions with the new schema... > So I check the source code That admin.rb might miss the setting for this > argument . > htd.setMaxFileSize(JLong.valueOf(arg.delete(MAX_FILESIZE))) if > arg[MAX_FILESIZE] > htd.setReadOnly(JBoolean.valueOf(arg.delete(READONLY))) if arg[READONLY] > ... > So I think it may be a bug ,is it? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19340) SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell
[ https://issues.apache.org/jira/browse/HBASE-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281370#comment-16281370 ] zhaoyuan commented on HBASE-19340: -- :) [~chia7712] Please help me to review this patch > SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell > --- > > Key: HBASE-19340 > URL: https://issues.apache.org/jira/browse/HBASE-19340 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.6 >Reporter: zhaoyuan >Assignee: zhaoyuan > Fix For: 1.2.8 > > Attachments: HBASE-19340-branch-1.2.batch, > HBASE-19340-branch-1.2.batch, HBASE-19340-branch-1.3.batch > > > Recently I wanna try to alter the split policy for a table on my cluster > which version is 1.2.6 and as far as I know The SPLIT_POLICY is an attribute > of the HTable so I run the command in hbase shell console below. > alter 'tablex',SPLIT_POLICY => > 'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy' > However, It gave the information like this and I confused > Unknown argument ignored: SPLIT_POLICY > Updating all regions with the new schema... > So I check the source code That admin.rb might miss the setting for this > argument . > htd.setMaxFileSize(JLong.valueOf(arg.delete(MAX_FILESIZE))) if > arg[MAX_FILESIZE] > htd.setReadOnly(JBoolean.valueOf(arg.delete(READONLY))) if arg[READONLY] > ... > So I think it may be a bug ,is it? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HBASE-19439) Mark ShortCircuitMasterConnection with InterfaceAudience Private
[ https://issues.apache.org/jira/browse/HBASE-19439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John resolved HBASE-19439. Resolution: Fixed Hadoop Flags: Reviewed Thanks for the quick review. > Mark ShortCircuitMasterConnection with InterfaceAudience Private > - > > Key: HBASE-19439 > URL: https://issues.apache.org/jira/browse/HBASE-19439 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19439.patch > > > ShortCircuitMasterConnection to be Private - It is wrongly declared Public. > (HBASE-17745) > Even MasterKeepAliveConnection also to be Private. This having no > InterfaceAudience now. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19448) Replace StringBuffer with StringBuilder
[
https://issues.apache.org/jira/browse/HBASE-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281366#comment-16281366
]
Hadoop QA commented on HBASE-19448:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
59s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
29s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
58s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
61m 51s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 2.7.4 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 93m
26s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}180m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19448 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12900998/HBASE-19448.1.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 939968cba09d 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 97d17ae195 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/10278/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/10278/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was automatically generated.
> Replace StringBuffer with StringBuilder
> ---
>
> Key: HBASE-19448
> URL:
[jira] [Commented] (HBASE-19437) Batch operation can't handle the null result for Append/Increment
[
https://issues.apache.org/jira/browse/HBASE-19437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281361#comment-16281361
]
Hadoop QA commented on HBASE-19437:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
49s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
58s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
57s{color} | {color:red} hbase-server: The patch generated 1 new + 132
unchanged - 0 fixed = 133 total (was 132) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
4s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
46m 43s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 2.7.4 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 91m
37s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}157m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19437 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12900999/HBASE-19437.v0.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux ab161803e369 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 97d17ae195 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/10279/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/10279/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/10279/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was automatically
[jira] [Updated] (HBASE-19430) Remove the SettableTimestamp and SettableSequenceId
[
https://issues.apache.org/jira/browse/HBASE-19430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-19430:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Release Note: All the cells which are used in server side are of
ExtendedCell now.
Status: Resolved (was: Patch Available)
Push the patch to branch-2 and master.
The method {{setTimestamp(byte[])}} exists in the commit as it is used in the
write path. The loop of updating the timestamp of cells will cause a lot of
conversion if we remove the method.
Thanks for all reviews.
> Remove the SettableTimestamp and SettableSequenceId
> ---
>
> Key: HBASE-19430
> URL: https://issues.apache.org/jira/browse/HBASE-19430
> Project: HBase
> Issue Type: Task
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
> Fix For: 2.0.0-beta-2
>
> Attachments: HBASE-19430.v0.patch, HBASE-19430.v0.test.patch,
> HBASE-19430.v1.patch, HBASE-19430.v1.patch
>
>
> They are introduced by HBASE-11777 and HBASE-12082. Both of them are IA.LP
> and denoted with deprecated. To my way of thinking, we should remove them
> before 2.0 release.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-13819) Make RPC layer CellBlock buffer a DirectByteBuffer
[ https://issues.apache.org/jira/browse/HBASE-13819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281355#comment-16281355 ] Anoop Sam John commented on HBASE-13819: So u mean the NIO wont keep that DBB in its ThreadLocal based pool. Good.. I believe it pools latest 5 DBBs or so. Somewhere saw this number. Means we can have a way to say max how many DBB size can get pooled . But that is in ThreadLocal means we have to * this number with max handlers counts. So that will give what max DBB the NIO layer can hold. The BBPool of ours is bit diff BTW. The DBBs are all fixed sized. There wont be creation of any random sized DBB and pool that. Ya that was the case in older impl in 1.x. There we had some max size for the BB which can be pooled. In 2.0 it is simpler. All DBBs are fixed size and we have a max #DBBs to pool also. So we know the max size the BBPool can grow. When request comes, we read that into one more MORE DBBs as per the req size. That way we avoid random sized DBB create. Only when the req pace is more and we are out of DBBs in pool, we will end up making use of the NIO way of random DBB create. Because when we can not see a free DBB from pool (to read reqs into), we will make on demand on heap BB and pass that to NIO. Ya Netty am also try to learn fully abt these aspects. > Make RPC layer CellBlock buffer a DirectByteBuffer > -- > > Key: HBASE-13819 > URL: https://issues.apache.org/jira/browse/HBASE-13819 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13819.patch, HBASE-13819_branch-1.patch, > HBASE-13819_branch-1.patch, HBASE-13819_branch-1.patch, q.png > > > In RPC layer, when we make a cellBlock to put as RPC payload, we will make an > on heap byte buffer (via BoundedByteBufferPool). The pool will keep upto > certain number of buffers. This jira aims at testing possibility for making > this buffers off heap ones. (DBB) The advantages > 1. Unsafe based writes to off heap is faster than that to on heap. Now we are > not using unsafe based writes at all. Even if we add, DBB will be better > 2. When Cells are backed by off heap (HBASE-11425) off heap to off heap > writes will be better > 3. When checked the code in SocketChannel impl, if we pass a HeapByteBuffer > to the socket channel, it will create a temp DBB and copy data to there and > only DBBs will be moved to Sockets. If we make DBB 1st hand itself, we can > avoid this one more level of copying. > Will do different perf testing with changed and report back. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19357) Bucket cache no longer L2 for LRU cache
[ https://issues.apache.org/jira/browse/HBASE-19357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281347#comment-16281347 ] stack commented on HBASE-19357: --- Ok. Sounds like we are good for offheap vs onheap. No data for onheap vs file. Suggest we file follow-on to measure and then deal if really bad -- if the whole cluster slows down because meta lookups are 20% or 10x slower (If out of oscache, my guess is that it won't be far behind the offheap -- to be proven). > Bucket cache no longer L2 for LRU cache > --- > > Key: HBASE-19357 > URL: https://issues.apache.org/jira/browse/HBASE-19357 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19357.patch, HBASE-19357.patch, > HBASE-19357_V2.patch, HBASE-19357_V3.patch, HBASE-19357_V3.patch > > > When Bucket cache is used, by default we dont configure it as an L2 cache > alone. The default setting is combined mode ON where the data blocks to > Bucket cache and index/bloom blocks go to LRU cache. But there is a way to > turn this off and make LRU as L1 and Bucket cache as a victim handler for L1. > It will be just L2. > After the off heap read path optimization Bucket cache is no longer slower > compared to L1. We have test results on data sizes from 12 GB. The Alibaba > use case was also with 12 GB and they have observed a ~30% QPS improve over > the LRU cache. > This issue is to remove the option for combined mode = false. So when Bucket > cache is in use, data blocks will go to it only and LRU will get only index > /meta/bloom blocks. Bucket cache will no longer be configured as a victim > handler for LRU. > Note : WHen external cache is in use, there only the L1 L2 thing comes. LRU > will be L1 and external cache act as its L2. That make full sense. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19341) Ensure CP can abort a Server
[ https://issues.apache.org/jira/browse/HBASE-19341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281342#comment-16281342 ] stack commented on HBASE-19341: --- Thanks for the correction [~anoop.hbase] > Ensure CP can abort a Server > > > Key: HBASE-19341 > URL: https://issues.apache.org/jira/browse/HBASE-19341 > Project: HBase > Issue Type: Improvement > Components: Coprocessors >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-1 > > > We used to allow a CP pull the Server#abort chain. We removed it in the CP > refactor. At the end of HBASE-18298, [~an...@apache.org] describes a case > where Phoenix needs to kill Server in extreme case to maintain consistency. > This issue is about ensuring that throwing a CPException will indeed kill the > running server Add a test. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19435) Reopen Files for ClosedChannelException in BucketCache
[ https://issues.apache.org/jira/browse/HBASE-19435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281339#comment-16281339 ] Ted Yu commented on HBASE-19435: Enclosing try / catch around single access call should be fine. > Reopen Files for ClosedChannelException in BucketCache > -- > > Key: HBASE-19435 > URL: https://issues.apache.org/jira/browse/HBASE-19435 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.0.0, 1.3.1 >Reporter: Zach York >Assignee: Zach York > Attachments: HBASE-19435.master.001.patch, > HBASE-19435.master.002.patch, HBASE-19435.master.003.patch, > HBASE-19435.master.004.patch, HBASE-19435.master.005.patch > > > When using the FileIOEngine for BucketCache, the cache will be disabled if > the connection is interrupted or closed. HBase will then get > ClosedChannelExceptions trying to access the file. After 60s, the RS will > disable the cache. This causes severe read performance degradation for > workloads that rely on this cache. FileIOEngine never tries to reopen the > connection. This JIRA is to reopen files when the BucketCache encounters a > ClosedChannelException. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19450) Add log about average execution time for ScheduledChore
Reid Chan created HBASE-19450: - Summary: Add log about average execution time for ScheduledChore Key: HBASE-19450 URL: https://issues.apache.org/jira/browse/HBASE-19450 Project: HBase Issue Type: Improvement Reporter: Reid Chan Assignee: Reid Chan Priority: Minor So far, there is no information about the exact execution time for a chore, we can provide log information about it. It also brings other benefits, like discovering inefficient chores which show rooms for improvement. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19341) Ensure CP can abort a Server
[ https://issues.apache.org/jira/browse/HBASE-19341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281329#comment-16281329 ] Anoop Sam John commented on HBASE-19341: bq.Hope this good by you Ashish Singhi. Not Ashish.. Concern from [~an...@apache.org] :-) U good with this Ankit? > Ensure CP can abort a Server > > > Key: HBASE-19341 > URL: https://issues.apache.org/jira/browse/HBASE-19341 > Project: HBase > Issue Type: Improvement > Components: Coprocessors >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-1 > > > We used to allow a CP pull the Server#abort chain. We removed it in the CP > refactor. At the end of HBASE-18298, [~an...@apache.org] describes a case > where Phoenix needs to kill Server in extreme case to maintain consistency. > This issue is about ensuring that throwing a CPException will indeed kill the > running server Add a test. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-19357) Bucket cache no longer L2 for LRU cache
[ https://issues.apache.org/jira/browse/HBASE-19357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281328#comment-16281328 ] Anoop Sam John edited comment on HBASE-19357 at 12/7/17 4:47 AM: - bq. It is years since we did heap vs BC compare. Did tests 2 weeks back also based on Alpha-4 release. This is again a single node test only comparing the on heap LRU cache (old L1) vs off heap Bucket Cache (Old L2) On heap Test -- Single server with 42 GB heap size. 60% cache size ie. 25 GB.Using G1GC with 65% as Initial Heap Occupancy Factor. Off heap Test -- Single server with 17 GB heap size and 25 GB off heap Bucket cache size. Here it is combined cache any way with just 20% as the L1 cache size. Again GC tuning is same PE tests with single CF and single column and 1K value size per cell(PE defaults). Pushed ~25GB data. Warming up cache before the read tests so as to put whole data in cache. Random row read test with 100 threads. Result On heap test gives a QPS of 421940.9 whereas off heap gives a QPS of 476190.5. So for 100 threads load we have a QPS gain of 12.8%. Did not compare on heap LRU vs File mode BC. We sure it will be slower but not data now on how slower. Only on File mode, I have a concern of META data not in on heap LRU. When it is off heap there is no concern. We sure the perf wont be bad. Did many test even on smaller data size like 16GB. The Alibaba tests (and cluster) is with 12 GB BC size off heap per server. The compare was on heap LRU vs off heap BC and they saw ~25% QPS gain at that time. was (Author: anoop.hbase): bq. It is years since we did heap vs BC compare. Did tests 2 weeks back also based on Alpha-4 release. This is again a single node test only comparing the on heap LRU cache (old L1) vs off heap Bucket Cache (Old L2) On heap Test -- Single server with 42 GB heap size. 60% cache size ie. 25 GB.Using G1GC with 65% as Initial Heap Occupancy Factor. Off heap Test -- Single server with 17 GB heap size and 25 GB off heap Bucket cache size. Here it is combined cache any way with just 20% as the L1 cache size. Again GC tuning is same PE tests with single CF and single column and 1K value size per cell(PE defaults). Pushed ~25GB data. Warming up cache before the read tests so as to put whole data in cache. Random row read test with 100 threads. Result On heap test gives a QPS of 421940.9 whereas off heap gives a QPS of 476190.5. So for 100 threads load we have a QPS gain of 12.8%. > Bucket cache no longer L2 for LRU cache > --- > > Key: HBASE-19357 > URL: https://issues.apache.org/jira/browse/HBASE-19357 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19357.patch, HBASE-19357.patch, > HBASE-19357_V2.patch, HBASE-19357_V3.patch, HBASE-19357_V3.patch > > > When Bucket cache is used, by default we dont configure it as an L2 cache > alone. The default setting is combined mode ON where the data blocks to > Bucket cache and index/bloom blocks go to LRU cache. But there is a way to > turn this off and make LRU as L1 and Bucket cache as a victim handler for L1. > It will be just L2. > After the off heap read path optimization Bucket cache is no longer slower > compared to L1. We have test results on data sizes from 12 GB. The Alibaba > use case was also with 12 GB and they have observed a ~30% QPS improve over > the LRU cache. > This issue is to remove the option for combined mode = false. So when Bucket > cache is in use, data blocks will go to it only and LRU will get only index > /meta/bloom blocks. Bucket cache will no longer be configured as a victim > handler for LRU. > Note : WHen external cache is in use, there only the L1 L2 thing comes. LRU > will be L1 and external cache act as its L2. That make full sense. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19357) Bucket cache no longer L2 for LRU cache
[ https://issues.apache.org/jira/browse/HBASE-19357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281328#comment-16281328 ] Anoop Sam John commented on HBASE-19357: bq. It is years since we did heap vs BC compare. Did tests 2 weeks back also based on Alpha-4 release. This is again a single node test only comparing the on heap LRU cache (old L1) vs off heap Bucket Cache (Old L2) On heap Test -- Single server with 42 GB heap size. 60% cache size ie. 25 GB.Using G1GC with 65% as Initial Heap Occupancy Factor. Off heap Test -- Single server with 17 GB heap size and 25 GB off heap Bucket cache size. Here it is combined cache any way with just 20% as the L1 cache size. Again GC tuning is same PE tests with single CF and single column and 1K value size per cell(PE defaults). Pushed ~25GB data. Warming up cache before the read tests so as to put whole data in cache. Random row read test with 100 threads. Result On heap test gives a QPS of 421940.9 whereas off heap gives a QPS of 476190.5. So for 100 threads load we have a QPS gain of 12.8%. > Bucket cache no longer L2 for LRU cache > --- > > Key: HBASE-19357 > URL: https://issues.apache.org/jira/browse/HBASE-19357 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19357.patch, HBASE-19357.patch, > HBASE-19357_V2.patch, HBASE-19357_V3.patch, HBASE-19357_V3.patch > > > When Bucket cache is used, by default we dont configure it as an L2 cache > alone. The default setting is combined mode ON where the data blocks to > Bucket cache and index/bloom blocks go to LRU cache. But there is a way to > turn this off and make LRU as L1 and Bucket cache as a victim handler for L1. > It will be just L2. > After the off heap read path optimization Bucket cache is no longer slower > compared to L1. We have test results on data sizes from 12 GB. The Alibaba > use case was also with 12 GB and they have observed a ~30% QPS improve over > the LRU cache. > This issue is to remove the option for combined mode = false. So when Bucket > cache is in use, data blocks will go to it only and LRU will get only index > /meta/bloom blocks. Bucket cache will no longer be configured as a victim > handler for LRU. > Note : WHen external cache is in use, there only the L1 L2 thing comes. LRU > will be L1 and external cache act as its L2. That make full sense. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19435) Reopen Files for ClosedChannelException in BucketCache
[ https://issues.apache.org/jira/browse/HBASE-19435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281321#comment-16281321 ] Anoop Sam John commented on HBASE-19435: BTW why buffer.limit(bufLimit); is included within try? This will never throw ClosedChannelException.. May be single line of access file can be in try? Any specific reason why kept this way or just as is? > Reopen Files for ClosedChannelException in BucketCache > -- > > Key: HBASE-19435 > URL: https://issues.apache.org/jira/browse/HBASE-19435 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.0.0, 1.3.1 >Reporter: Zach York >Assignee: Zach York > Attachments: HBASE-19435.master.001.patch, > HBASE-19435.master.002.patch, HBASE-19435.master.003.patch, > HBASE-19435.master.004.patch, HBASE-19435.master.005.patch > > > When using the FileIOEngine for BucketCache, the cache will be disabled if > the connection is interrupted or closed. HBase will then get > ClosedChannelExceptions trying to access the file. After 60s, the RS will > disable the cache. This causes severe read performance degradation for > workloads that rely on this cache. FileIOEngine never tries to reopen the > connection. This JIRA is to reopen files when the BucketCache encounters a > ClosedChannelException. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18962) Support atomic BatchOperations through batchMutate()
[ https://issues.apache.org/jira/browse/HBASE-18962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281323#comment-16281323 ] Umesh Agashe commented on HBASE-18962: -- Thanks [~chancelq]! Great if you could add UT. Let me know if you create JIRA for it. > Support atomic BatchOperations through batchMutate() > > > Key: HBASE-18962 > URL: https://issues.apache.org/jira/browse/HBASE-18962 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.0-alpha-3 >Reporter: Umesh Agashe >Assignee: Umesh Agashe > Fix For: 2.0.0-beta-1 > > Attachments: hbase-18962.master.001.patch, > hbase-18962.master.002.patch, hbase-18962.master.003.patch, > hbase-18962.master.004.patch > > > Support all mutations in BatchOperations to be applied atomically (all or > none) by locking all rows corresponding to mutations exclusively. > mutateRows() which uses MultiRowMutationProcessor applies all mutations > atomically and batchMutate() is non-atomic. To unify code paths, isAtomic() > attribute can be added to BatchOperations. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19435) Reopen Files for ClosedChannelException in BucketCache
[
https://issues.apache.org/jira/browse/HBASE-19435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281322#comment-16281322
]
Hadoop QA commented on HBASE-19435:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 8m
42s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
47s{color} | {color:red} hbase-server: The patch generated 2 new + 11 unchanged
- 0 fixed = 13 total (was 11) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 7m
4s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
78m 21s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 2.7.4 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}172m 10s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}284m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.security.access.TestCoprocessorWhitelistMasterObserver |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19435 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12900977/HBASE-19435.master.002.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 1658de4e124c 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 4a2e8b852d |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/10274/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/10274/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
[jira] [Commented] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
[
https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281320#comment-16281320
]
Hadoop QA commented on HBASE-19389:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
51s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
56s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
32s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
14s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} The patch hbase-common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s{color} | {color:green} hbase-server: The patch generated 0 new + 403
unchanged - 1 fixed = 403 total (was 404) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
44s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
40m 23s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 2.7.4 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
12s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}107m 52s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}174m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:9f2f2db |
| JIRA Issue | HBASE-19389 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12900995/HBASE-19389-branch-2-V4.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 98a1a1320fa0 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git
[jira] [Commented] (HBASE-19447) INFO level logging of GetClusterStatus from HMaster is too chatty
[ https://issues.apache.org/jira/browse/HBASE-19447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281318#comment-16281318 ] Hudson commented on HBASE-19447: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4182 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4182/]) HBASE-19447 INFO level logging of GetClusterStatus from HMaster is too (apurtell: rev 97d17ae195cd522d0ce74db0733f63337a36b480) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java > INFO level logging of GetClusterStatus from HMaster is too chatty > - > > Key: HBASE-19447 > URL: https://issues.apache.org/jira/browse/HBASE-19447 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0 > > > Should be DEBUG. Should be wrapped with a logLevel check. > ./hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java:2409: > LOG.info(getClientIdAuditPrefix() + " get ClusterStatus, status=" + > status); -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19445) PerformanceEvaluation NPE processing split policy option
[ https://issues.apache.org/jira/browse/HBASE-19445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281316#comment-16281316 ] Hudson commented on HBASE-19445: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4182 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4182/]) HBASE-19445 PerformanceEvaluation NPE processing split policy option (apurtell: rev 3e7b90ac6d808c171ea988d8d32ef998146713ac) * (edit) hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java > PerformanceEvaluation NPE processing split policy option > > > Key: HBASE-19445 > URL: https://issues.apache.org/jira/browse/HBASE-19445 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Trivial > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-19445-branch-1.patch, HBASE-19445.patch > > > $ ./bin/hbase pe --nomapred randomWrite 10 > java.lang.NullPointerException > at > org.apache.hadoop.hbase.PerformanceEvaluation.getTableDescriptor(PerformanceEvaluation.java:387) > at > org.apache.hadoop.hbase.PerformanceEvaluation.checkTable(PerformanceEvaluation.java:356) > at > org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:1778) > at > org.apache.hadoop.hbase.PerformanceEvaluation.run(PerformanceEvaluation.java:2168) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at > org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:2189) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19295) The Configuration returned by CPEnv should be read-only.
[ https://issues.apache.org/jira/browse/HBASE-19295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281317#comment-16281317 ] Hudson commented on HBASE-19295: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4182 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4182/]) HBASE-19295 The Configuration returned by CPEnv should be read-only. (stack: rev 4a2e8b852dceb8f68adf33c940621734cc6e2f12) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/CoprocessorEnvironment.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseEnvironment.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/ReadOnlyConfiguration.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorConfiguration.java > The Configuration returned by CPEnv should be read-only. > > > Key: HBASE-19295 > URL: https://issues.apache.org/jira/browse/HBASE-19295 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: stack >Assignee: stack > Labels: incompatible > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19295.master.001.patch, > HBASE-19295.master.002.patch, HBASE-19295.master.003.patch, > HBASE-19295.master.004.patch, HBASE-19295.master.005.patch > > > The Configuration a CP gets when it does a getConfiguration on the > environment is that of the RegionServer. The CP should not be able to modify > this config. We should throw exception if they try to write us. > Ditto w/ the Connection they can get from the env. They should not be able to > close it at min. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19435) Reopen Files for ClosedChannelException in BucketCache
[ https://issues.apache.org/jira/browse/HBASE-19435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281314#comment-16281314 ] Anoop Sam John commented on HBASE-19435: Thanks Zack I see u say abt this config hbase.bucketcache.ioengine.errors.tolerated.duration and the def is 1 min. Fine to handle enable cache again after some time in another issue.. Ya seems we must handle it. Would be good to add a TODO abt that work here. (If u will open another jira now itself it is fine that way also :-) ) +1 > Reopen Files for ClosedChannelException in BucketCache > -- > > Key: HBASE-19435 > URL: https://issues.apache.org/jira/browse/HBASE-19435 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.0.0, 1.3.1 >Reporter: Zach York >Assignee: Zach York > Attachments: HBASE-19435.master.001.patch, > HBASE-19435.master.002.patch, HBASE-19435.master.003.patch, > HBASE-19435.master.004.patch, HBASE-19435.master.005.patch > > > When using the FileIOEngine for BucketCache, the cache will be disabled if > the connection is interrupted or closed. HBase will then get > ClosedChannelExceptions trying to access the file. After 60s, the RS will > disable the cache. This causes severe read performance degradation for > workloads that rely on this cache. FileIOEngine never tries to reopen the > connection. This JIRA is to reopen files when the BucketCache encounters a > ClosedChannelException. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19435) Reopen Files for ClosedChannelException in BucketCache
[
https://issues.apache.org/jira/browse/HBASE-19435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281308#comment-16281308
]
Hadoop QA commented on HBASE-19435:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
54s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
37s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
4s{color} | {color:red} hbase-server: The patch generated 1 new + 10 unchanged
- 1 fixed = 11 total (was 11) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
31s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
54m 15s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 2.7.4 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}116m
16s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}192m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19435 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12900991/HBASE-19435.master.005.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 4a00a142c8fb 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 4a2e8b852d |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/10275/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/10275/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/10275/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was
[jira] [Commented] (HBASE-19340) SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell
[
https://issues.apache.org/jira/browse/HBASE-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281307#comment-16281307
]
Hadoop QA commented on HBASE-19340:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
1s{color} | {color:blue} The patch file was not named according to hbase's
naming conventions. Please see
https://yetus.apache.org/documentation/0.6.0/precommit-patchnames for
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-1.3 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
40s{color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
6s{color} | {color:green} branch-1.3 passed with JDK v1.8.0_152 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} branch-1.3 passed with JDK v1.7.0_161 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m
9s{color} | {color:red} The patch generated 67 new + 943 unchanged - 27 fixed =
1010 total (was 970) {color} |
| {color:red}-1{color} | {color:red} ruby-lint {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 157 new + 876 unchanged - 8 fixed
= 1033 total (was 884) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
7s{color} | {color:green} the patch passed with JDK v1.8.0_152 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} the patch passed with JDK v1.7.0_161 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m
43s{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
7s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:dca6535 |
| JIRA Issue | HBASE-19340 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12901001/HBASE-19340-branch-1.3.batch
|
| Optional Tests | asflicense javac javadoc unit rubocop ruby_lint |
| uname | Linux f9e7f520ad72 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | branch-1.3 / 6d8c080 |
| maven | version: Apache Maven 3.0.5 |
| Default Java | 1.7.0_161 |
| Multi-JDK versions | /usr/lib/jvm/java-8-openjdk-amd64:1.8.0_152
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_161 |
| rubocop | v0.51.0 |
| rubocop |
https://builds.apache.org/job/PreCommit-HBASE-Build/10280/artifact/patchprocess/diff-patch-rubocop.txt
|
| ruby-lint | v2.3.1 |
| ruby-lint |
https://builds.apache.org/job/PreCommit-HBASE-Build/10280/artifact/patchprocess/diff-patch-ruby-lint.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/10280/testReport/ |
| modules | C: hbase-shell U: hbase-shell |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/10280/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was automatically generated.
> SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell
> ---
>
> Key: HBASE-19340
> URL: https://issues.apache.org/jira/browse/HBASE-19340
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.6
>Reporter: zhaoyuan
>Assignee: zhaoyuan
> Fix For: 1.2.8
>
>
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[
https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281305#comment-16281305
]
stack commented on HBASE-16890:
---
Comparing one client with a few threads (6) against a remote, single
regionserver that is running on a HDFS of seven datanodes, I see async as
slower. For workload a (50/50) its 4633 for old WAL vs 2812 for async
ops/second.
echo "`date` 50/50 workload a"
time "${YCSB}" run "${BINDING}" -P "${WORKLOADS}/workloada"
"${WORKLOAD_ARGS[@]}" -p recordcount=0 -p operationcount=${INSERT_COUNT} \
-p exportfile="${LOGS}/ycsb-workloada-measurements-${HERE}-${date}.json" -p
exporter="${EXPORTER}" \
>"${LOGS}/ycsb-workloada-${HERE}-${date}.out"
2>"${LOGS}/ycsb-workloada-${HERE}-${date}.err"
INSERT_COUNT=1
TIME=3600
6 Clients/Threads.
I've not spent any time digging in on why the big diff. Pasting this here while
I rerun my test and do a bit of study as it runs...
> Analyze the performance of AsyncWAL and fix the same
>
>
> Key: HBASE-16890
> URL: https://issues.apache.org/jira/browse/HBASE-16890
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Blocker
> Fix For: 2.0.0-beta-1
>
> Attachments: AsyncWAL_disruptor.patch, AsyncWAL_disruptor_1
> (2).patch, AsyncWAL_disruptor_3.patch, AsyncWAL_disruptor_3.patch,
> AsyncWAL_disruptor_4.patch, AsyncWAL_disruptor_6.patch,
> HBASE-16890-rc-v2.patch, HBASE-16890-rc-v3.patch,
> HBASE-16890-remove-contention-v1.patch, HBASE-16890-remove-contention.patch,
> Screen Shot 2016-10-25 at 7.34.47 PM.png, Screen Shot 2016-10-25 at 7.39.07
> PM.png, Screen Shot 2016-10-25 at 7.39.48 PM.png, Screen Shot 2016-11-04 at
> 5.21.27 PM.png, Screen Shot 2016-11-04 at 5.30.18 PM.png, async.svg,
> classic.svg, contention.png, contention_defaultWAL.png,
> ycsb_FSHlog.vs.Async.png
>
>
> Tests reveal that AsyncWAL under load in single node cluster performs slower
> than the Default WAL. This task is to analyze and see if we could fix it.
> See some discussions in the tail of JIRA HBASE-15536.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-15482) Provide an option to skip calculating block locations for SnapshotInputFormat
[ https://issues.apache.org/jira/browse/HBASE-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiang Li updated HBASE-15482: - Status: Open (was: Patch Available) > Provide an option to skip calculating block locations for SnapshotInputFormat > - > > Key: HBASE-15482 > URL: https://issues.apache.org/jira/browse/HBASE-15482 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Liyin Tang >Assignee: Xiang Li >Priority: Minor > Fix For: 2.1.0 > > Attachments: HBASE-15482.master.000.patch > > > When a MR job is reading from SnapshotInputFormat, it needs to calculate the > splits based on the block locations in order to get best locality. However, > this process may take a long time for large snapshots. > In some setup, the computing layer, Spark, Hive or Presto could run out side > of HBase cluster. In these scenarios, the block locality doesn't matter. > Therefore, it will be great to have an option to skip calculating the block > locations for every job. That will super useful for the Hive/Presto/Spark > connectors. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-15482) Provide an option to skip calculating block locations for SnapshotInputFormat
[
https://issues.apache.org/jira/browse/HBASE-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281299#comment-16281299
]
Ted Yu commented on HBASE-15482:
{code}
+ public static final String SNAPSHOT_INPUTFORMAT_CARE_BLOCK_LOCALITY_KEY =
"hbase.TableSnapshotInputFormat.locality";
{code}
>From the current key name, it sounds like locality measure (say percent). How
>about naming it "hbase.TableSnapshotInputFormat.locality.enable" (or something
>similar).
{code}
+ List hosts = getBestLocations(conf,
+ HRegion.computeHDFSBlocksDistribution(conf, htd, hri, tableDir));
+
+ // return at most top 3 hosts
+ int len = Math.min(3, hosts.size());
{code}
You can pass 3 to getBestLocations() so that no more than 3 hosts are returned.
> Provide an option to skip calculating block locations for SnapshotInputFormat
> -
>
> Key: HBASE-15482
> URL: https://issues.apache.org/jira/browse/HBASE-15482
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce
>Reporter: Liyin Tang
>Assignee: Xiang Li
>Priority: Minor
> Fix For: 2.1.0
>
> Attachments: HBASE-15482.master.000.patch
>
>
> When a MR job is reading from SnapshotInputFormat, it needs to calculate the
> splits based on the block locations in order to get best locality. However,
> this process may take a long time for large snapshots.
> In some setup, the computing layer, Spark, Hive or Presto could run out side
> of HBase cluster. In these scenarios, the block locality doesn't matter.
> Therefore, it will be great to have an option to skip calculating the block
> locations for every job. That will super useful for the Hive/Presto/Spark
> connectors.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HBASE-19449) Implement SLF4J and SLF4J Parameter Substitution
[ https://issues.apache.org/jira/browse/HBASE-19449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR reassigned HBASE-19449: --- Assignee: BELUGA BEHR > Implement SLF4J and SLF4J Parameter Substitution > > > Key: HBASE-19449 > URL: https://issues.apache.org/jira/browse/HBASE-19449 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 2.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-19449.1.patch > > > * Use SLF4J logging > * Use SLF4J parameter substitution > * Fix some small issues with missing spaces between words in the log message > and the like > https://www.slf4j.org/faq.html#logging_performance -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19449) Implement SLF4J and SLF4J Parameter Substitution
[
https://issues.apache.org/jira/browse/HBASE-19449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HBASE-19449:
Description:
For the {{HFileArchiver.java}} class...
* Use SLF4J logging
* Use SLF4J parameter substitution
* Fix some small issues with missing spaces between words in the log message
and the like
https://www.slf4j.org/faq.html#logging_performance
was:
* Use SLF4J logging
* Use SLF4J parameter substitution
* Fix some small issues with missing spaces between words in the log message
and the like
https://www.slf4j.org/faq.html#logging_performance
> Implement SLF4J and SLF4J Parameter Substitution
>
>
> Key: HBASE-19449
> URL: https://issues.apache.org/jira/browse/HBASE-19449
> Project: HBase
> Issue Type: Improvement
> Components: hbase
>Affects Versions: 2.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Fix For: 2.0.0
>
> Attachments: HBASE-19449.1.patch
>
>
> For the {{HFileArchiver.java}} class...
> * Use SLF4J logging
> * Use SLF4J parameter substitution
> * Fix some small issues with missing spaces between words in the log message
> and the like
> https://www.slf4j.org/faq.html#logging_performance
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19449) Implement SLF4J and SLF4J Parameter Substitution
[ https://issues.apache.org/jira/browse/HBASE-19449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HBASE-19449: Status: Patch Available (was: Open) > Implement SLF4J and SLF4J Parameter Substitution > > > Key: HBASE-19449 > URL: https://issues.apache.org/jira/browse/HBASE-19449 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 2.0.0 >Reporter: BELUGA BEHR >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-19449.1.patch > > > * Use SLF4J logging > * Use SLF4J parameter substitution > * Fix some small issues with missing spaces between words in the log message > and the like > https://www.slf4j.org/faq.html#logging_performance -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19449) Implement SLF4J and SLF4J Parameter Substitution
BELUGA BEHR created HBASE-19449: --- Summary: Implement SLF4J and SLF4J Parameter Substitution Key: HBASE-19449 URL: https://issues.apache.org/jira/browse/HBASE-19449 Project: HBase Issue Type: Improvement Components: hbase Affects Versions: 2.0.0 Reporter: BELUGA BEHR Priority: Trivial Fix For: 2.0.0 * Use SLF4J logging * Use SLF4J parameter substitution * Fix some small issues with missing spaces between words in the log message and the like https://www.slf4j.org/faq.html#logging_performance -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19449) Implement SLF4J and SLF4J Parameter Substitution
[ https://issues.apache.org/jira/browse/HBASE-19449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HBASE-19449: Attachment: HBASE-19449.1.patch > Implement SLF4J and SLF4J Parameter Substitution > > > Key: HBASE-19449 > URL: https://issues.apache.org/jira/browse/HBASE-19449 > Project: HBase > Issue Type: Improvement > Components: hbase >Affects Versions: 2.0.0 >Reporter: BELUGA BEHR >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-19449.1.patch > > > * Use SLF4J logging > * Use SLF4J parameter substitution > * Fix some small issues with missing spaces between words in the log message > and the like > https://www.slf4j.org/faq.html#logging_performance -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19340) SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell
[ https://issues.apache.org/jira/browse/HBASE-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaoyuan updated HBASE-19340: - Attachment: HBASE-19340-branch-1.3.batch > SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell > --- > > Key: HBASE-19340 > URL: https://issues.apache.org/jira/browse/HBASE-19340 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.6 >Reporter: zhaoyuan >Assignee: zhaoyuan > Fix For: 1.2.8 > > Attachments: HBASE-19340-branch-1.2.batch, > HBASE-19340-branch-1.2.batch, HBASE-19340-branch-1.3.batch > > > Recently I wanna try to alter the split policy for a table on my cluster > which version is 1.2.6 and as far as I know The SPLIT_POLICY is an attribute > of the HTable so I run the command in hbase shell console below. > alter 'tablex',SPLIT_POLICY => > 'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy' > However, It gave the information like this and I confused > Unknown argument ignored: SPLIT_POLICY > Updating all regions with the new schema... > So I check the source code That admin.rb might miss the setting for this > argument . > htd.setMaxFileSize(JLong.valueOf(arg.delete(MAX_FILESIZE))) if > arg[MAX_FILESIZE] > htd.setReadOnly(JBoolean.valueOf(arg.delete(READONLY))) if arg[READONLY] > ... > So I think it may be a bug ,is it? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19439) Mark ShortCircuitMasterConnection with InterfaceAudience Private
[ https://issues.apache.org/jira/browse/HBASE-19439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281295#comment-16281295 ] Yu Li commented on HBASE-19439: --- +1 > Mark ShortCircuitMasterConnection with InterfaceAudience Private > - > > Key: HBASE-19439 > URL: https://issues.apache.org/jira/browse/HBASE-19439 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0-alpha-1 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19439.patch > > > ShortCircuitMasterConnection to be Private - It is wrongly declared Public. > (HBASE-17745) > Even MasterKeepAliveConnection also to be Private. This having no > InterfaceAudience now. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HBASE-19448) Replace StringBuffer with StringBuilder
[
https://issues.apache.org/jira/browse/HBASE-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai reassigned HBASE-19448:
--
Assignee: BELUGA BEHR
> Replace StringBuffer with StringBuilder
> ---
>
> Key: HBASE-19448
> URL: https://issues.apache.org/jira/browse/HBASE-19448
> Project: HBase
> Issue Type: Improvement
> Components: hbase
>Affects Versions: 2.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Fix For: 2.0.0
>
> Attachments: HBASE-19448.1.patch
>
>
> Replace the thread-safe {{StringBuffer}} class with the un-synchronized
> {{StringBuilder}} class for better performance.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19448) Replace StringBuffer with StringBuilder
[
https://issues.apache.org/jira/browse/HBASE-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Ping Tsai updated HBASE-19448:
---
Fix Version/s: 2.0.0
> Replace StringBuffer with StringBuilder
> ---
>
> Key: HBASE-19448
> URL: https://issues.apache.org/jira/browse/HBASE-19448
> Project: HBase
> Issue Type: Improvement
> Components: hbase
>Affects Versions: 2.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Fix For: 2.0.0
>
> Attachments: HBASE-19448.1.patch
>
>
> Replace the thread-safe {{StringBuffer}} class with the un-synchronized
> {{StringBuilder}} class for better performance.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19448) Replace StringBuffer with StringBuilder
[
https://issues.apache.org/jira/browse/HBASE-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281288#comment-16281288
]
Chia-Ping Tsai commented on HBASE-19448:
LGTM. I do the grep for StringBuffer. Seems the patch doesn't do the
replacement totally. Could you take a look? [~belugabehr] Thanks
> Replace StringBuffer with StringBuilder
> ---
>
> Key: HBASE-19448
> URL: https://issues.apache.org/jira/browse/HBASE-19448
> Project: HBase
> Issue Type: Improvement
> Components: hbase
>Affects Versions: 2.0.0
>Reporter: BELUGA BEHR
>Priority: Trivial
> Attachments: HBASE-19448.1.patch
>
>
> Replace the thread-safe {{StringBuffer}} class with the un-synchronized
> {{StringBuilder}} class for better performance.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19442) Backport HBASE-19065 to branch-1 (HRegion#bulkLoadHFiles() should wait for concurrent Region#flush() to finish)
[ https://issues.apache.org/jira/browse/HBASE-19442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281283#comment-16281283 ] Hudson commented on HBASE-19442: FAILURE: Integrated in Jenkins build HBase-1.4 #1054 (See [https://builds.apache.org/job/HBase-1.4/1054/]) HBASE-19442 Backport HBASE-19065 to branch-1 (HRegion#bulkLoadHFiles() (apurtell: rev fda742b35b58cdac40e4b0c1891625346a2f9d47) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > Backport HBASE-19065 to branch-1 (HRegion#bulkLoadHFiles() should wait for > concurrent Region#flush() to finish) > --- > > Key: HBASE-19442 > URL: https://issues.apache.org/jira/browse/HBASE-19442 > Project: HBase > Issue Type: Bug >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar > Fix For: 1.4.0, 1.3.2, 1.2.7 > > Attachments: HBASE-19442-branch-1.patch > > > This JIRA is to backport HBASE-19065 to branch-1/1.4/1.3/1.2. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19065) HRegion#bulkLoadHFiles() should wait for concurrent Region#flush() to finish
[
https://issues.apache.org/jira/browse/HBASE-19065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281284#comment-16281284
]
Hudson commented on HBASE-19065:
FAILURE: Integrated in Jenkins build HBase-1.4 #1054 (See
[https://builds.apache.org/job/HBase-1.4/1054/])
HBASE-19442 Backport HBASE-19065 to branch-1 (HRegion#bulkLoadHFiles()
(apurtell: rev fda742b35b58cdac40e4b0c1891625346a2f9d47)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> HRegion#bulkLoadHFiles() should wait for concurrent Region#flush() to finish
>
>
> Key: HBASE-19065
> URL: https://issues.apache.org/jira/browse/HBASE-19065
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0-beta-1
>
> Attachments: 19065.v1.txt, 19065.v2.txt, 19065.v2.txt
>
>
> When I was debugging bulk load failure, I saw the following in region server
> log:
> {code}
> 2017-10-17 23:05:28,795 DEBUG
> [B.defaultRpcServer.handler=0,queue=0,port=16020] regionserver.HRegion: NOT
> flushing memstore for region mx_,
> f449669a8b0341e4edbd2ebdacc72094f449669a8b0341e4edbd2ebdacc7209420150711,1504909319142.52d496ba39036e0c2cc9522895ad438f.,
> flushing=true, writesEnabled=true
> 2017-10-17 23:05:28,796 ERROR
> [B.defaultRpcServer.handler=0,queue=0,port=16020]
> access.SecureBulkLoadEndpoint: Failed to complete bulk load
> java.io.IOException: Could not bulk load with an assigned sequential ID
> because the flush didn't run. Reason for not flushing: Not flushing since
> already flushing
> at
> org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:5282)
> at
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint$1.run(SecureBulkLoadEndpoint.java:292)
> at
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint$1.run(SecureBulkLoadEndpoint.java:275)
> {code}
> There was concurrent flush which got misinterpreted by bulkLoadHFiles().
> HRegion#bulkLoadHFiles() should wait for the concurrent flush to complete.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19440) Not able to enable balancer with RSGroups once disabled
[ https://issues.apache.org/jira/browse/HBASE-19440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281282#comment-16281282 ] Hudson commented on HBASE-19440: FAILURE: Integrated in Jenkins build HBase-1.4 #1054 (See [https://builds.apache.org/job/HBase-1.4/1054/]) HBASE-19440 Not able to enable balancer with RSGroups once disabled (apurtell: rev 48b41c4b13fc4208d4f360e388a66dbe0ce533e3) * (edit) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroups.java * (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java > Not able to enable balancer with RSGroups once disabled > --- > > Key: HBASE-19440 > URL: https://issues.apache.org/jira/browse/HBASE-19440 > Project: HBase > Issue Type: Bug >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Fix For: 1.4.0 > > Attachments: HBASE-19440.branch-1.001.patch > > > Once the balancer is disabled, trying to switch it back on doesn't work since > the prebalanceswitch coprocessor hook is incorrectly always returning false. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19437) Batch operation can't handle the null result for Append/Increment
[ https://issues.apache.org/jira/browse/HBASE-19437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-19437: --- Attachment: HBASE-19437.v0.patch retry > Batch operation can't handle the null result for Append/Increment > - > > Key: HBASE-19437 > URL: https://issues.apache.org/jira/browse/HBASE-19437 > Project: HBase > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai > Fix For: 2.0.0 > > Attachments: HBASE-19437.v0.patch, HBASE-19437.v0.patch > > > But the Table#append and #increment can handle the null result...that is an > inconsistent behavior for user. > I have noticed two scenarios that server will return null result to user. > # postAppend/postIncrement return null > # mutation.isReturnResults() is false and > preIncrementAfterRowLock/preAppendAfterRowLock doesn't return null > We should wrap the null to empty result on server side. CP user should throw > Exception rather than return null if they intend to say something is broken. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19448) Replace StringBuffer with StringBuilder
[
https://issues.apache.org/jira/browse/HBASE-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281260#comment-16281260
]
Ted Yu commented on HBASE-19448:
+1
> Replace StringBuffer with StringBuilder
> ---
>
> Key: HBASE-19448
> URL: https://issues.apache.org/jira/browse/HBASE-19448
> Project: HBase
> Issue Type: Improvement
> Components: hbase
>Affects Versions: 2.0.0
>Reporter: BELUGA BEHR
>Priority: Trivial
> Attachments: HBASE-19448.1.patch
>
>
> Replace the thread-safe {{StringBuffer}} class with the un-synchronized
> {{StringBuilder}} class for better performance.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19448) Replace StringBuffer with StringBuilder
[
https://issues.apache.org/jira/browse/HBASE-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HBASE-19448:
Status: Patch Available (was: Open)
> Replace StringBuffer with StringBuilder
> ---
>
> Key: HBASE-19448
> URL: https://issues.apache.org/jira/browse/HBASE-19448
> Project: HBase
> Issue Type: Improvement
> Components: hbase
>Affects Versions: 2.0.0
>Reporter: BELUGA BEHR
>Priority: Trivial
> Attachments: HBASE-19448.1.patch
>
>
> Replace the thread-safe {{StringBuffer}} class with the un-synchronized
> {{StringBuilder}} class for better performance.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19448) Replace StringBuffer with StringBuilder
[
https://issues.apache.org/jira/browse/HBASE-19448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HBASE-19448:
Attachment: HBASE-19448.1.patch
> Replace StringBuffer with StringBuilder
> ---
>
> Key: HBASE-19448
> URL: https://issues.apache.org/jira/browse/HBASE-19448
> Project: HBase
> Issue Type: Improvement
> Components: hbase
>Affects Versions: 2.0.0
>Reporter: BELUGA BEHR
>Priority: Trivial
> Attachments: HBASE-19448.1.patch
>
>
> Replace the thread-safe {{StringBuffer}} class with the un-synchronized
> {{StringBuilder}} class for better performance.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HBASE-19448) Replace StringBuffer with StringBuilder
BELUGA BEHR created HBASE-19448:
---
Summary: Replace StringBuffer with StringBuilder
Key: HBASE-19448
URL: https://issues.apache.org/jira/browse/HBASE-19448
Project: HBase
Issue Type: Improvement
Components: hbase
Affects Versions: 2.0.0
Reporter: BELUGA BEHR
Priority: Trivial
Replace the thread-safe {{StringBuffer}} class with the un-synchronized
{{StringBuilder}} class for better performance.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15482) Provide an option to skip calculating block locations for SnapshotInputFormat
[
https://issues.apache.org/jira/browse/HBASE-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281248#comment-16281248
]
Hadoop QA commented on HBASE-15482:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
52s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
52s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} hbase-mapreduce: The patch generated 15 new + 17
unchanged - 0 fixed = 32 total (was 17) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
33s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
54m 6s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 2.7.4 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m
38s{color} | {color:green} hbase-mapreduce in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 84m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-15482 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12900899/HBASE-15482.master.000.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux eeaf111c9913 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 4a2e8b852d |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/10276/artifact/patchprocess/diff-checkstyle-hbase-mapreduce.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/10276/testReport/ |
| modules | C: hbase-mapreduce U: hbase-mapreduce |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/10276/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This
[jira] [Commented] (HBASE-19340) SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell
[ https://issues.apache.org/jira/browse/HBASE-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281242#comment-16281242 ] zhaoyuan commented on HBASE-19340: -- OK, let me check it out. > SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell > --- > > Key: HBASE-19340 > URL: https://issues.apache.org/jira/browse/HBASE-19340 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.6 >Reporter: zhaoyuan >Assignee: zhaoyuan > Fix For: 1.2.8 > > Attachments: HBASE-19340-branch-1.2.batch, > HBASE-19340-branch-1.2.batch > > > Recently I wanna try to alter the split policy for a table on my cluster > which version is 1.2.6 and as far as I know The SPLIT_POLICY is an attribute > of the HTable so I run the command in hbase shell console below. > alter 'tablex',SPLIT_POLICY => > 'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy' > However, It gave the information like this and I confused > Unknown argument ignored: SPLIT_POLICY > Updating all regions with the new schema... > So I check the source code That admin.rb might miss the setting for this > argument . > htd.setMaxFileSize(JLong.valueOf(arg.delete(MAX_FILESIZE))) if > arg[MAX_FILESIZE] > htd.setReadOnly(JBoolean.valueOf(arg.delete(READONLY))) if arg[READONLY] > ... > So I think it may be a bug ,is it? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19435) Reopen Files for ClosedChannelException in BucketCache
[ https://issues.apache.org/jira/browse/HBASE-19435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281236#comment-16281236 ] Ted Yu commented on HBASE-19435: lgtm > Reopen Files for ClosedChannelException in BucketCache > -- > > Key: HBASE-19435 > URL: https://issues.apache.org/jira/browse/HBASE-19435 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.0.0, 1.3.1 >Reporter: Zach York >Assignee: Zach York > Attachments: HBASE-19435.master.001.patch, > HBASE-19435.master.002.patch, HBASE-19435.master.003.patch, > HBASE-19435.master.004.patch, HBASE-19435.master.005.patch > > > When using the FileIOEngine for BucketCache, the cache will be disabled if > the connection is interrupted or closed. HBase will then get > ClosedChannelExceptions trying to access the file. After 60s, the RS will > disable the cache. This causes severe read performance degradation for > workloads that rely on this cache. FileIOEngine never tries to reopen the > connection. This JIRA is to reopen files when the BucketCache encounters a > ClosedChannelException. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19447) INFO level logging of GetClusterStatus from HMaster is too chatty
[ https://issues.apache.org/jira/browse/HBASE-19447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281233#comment-16281233 ] Chia-Ping Tsai commented on HBASE-19447: +1 > INFO level logging of GetClusterStatus from HMaster is too chatty > - > > Key: HBASE-19447 > URL: https://issues.apache.org/jira/browse/HBASE-19447 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0 > > > Should be DEBUG. Should be wrapped with a logLevel check. > ./hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java:2409: > LOG.info(getClientIdAuditPrefix() + " get ClusterStatus, status=" + > status); -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19445) PerformanceEvaluation NPE processing split policy option
[ https://issues.apache.org/jira/browse/HBASE-19445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281230#comment-16281230 ] Chia-Ping Tsai commented on HBASE-19445: +1 > PerformanceEvaluation NPE processing split policy option > > > Key: HBASE-19445 > URL: https://issues.apache.org/jira/browse/HBASE-19445 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Trivial > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-19445-branch-1.patch, HBASE-19445.patch > > > $ ./bin/hbase pe --nomapred randomWrite 10 > java.lang.NullPointerException > at > org.apache.hadoop.hbase.PerformanceEvaluation.getTableDescriptor(PerformanceEvaluation.java:387) > at > org.apache.hadoop.hbase.PerformanceEvaluation.checkTable(PerformanceEvaluation.java:356) > at > org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:1778) > at > org.apache.hadoop.hbase.PerformanceEvaluation.run(PerformanceEvaluation.java:2168) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at > org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:2189) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19340) SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell
[ https://issues.apache.org/jira/browse/HBASE-19340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281228#comment-16281228 ] Chia-Ping Tsai commented on HBASE-19340: bq. I have backported all missed options except PRIORITY The priority setting is introduced by HBASE-16095. It exists in branch-1.3+. Do you mind making the patch for branch-1.3? Just add the PRIORITY I believe. > SPLIT_POLICY and FLUSH_POLICY cann't be set directly by hbase shell > --- > > Key: HBASE-19340 > URL: https://issues.apache.org/jira/browse/HBASE-19340 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.6 >Reporter: zhaoyuan >Assignee: zhaoyuan > Fix For: 1.2.8 > > Attachments: HBASE-19340-branch-1.2.batch, > HBASE-19340-branch-1.2.batch > > > Recently I wanna try to alter the split policy for a table on my cluster > which version is 1.2.6 and as far as I know The SPLIT_POLICY is an attribute > of the HTable so I run the command in hbase shell console below. > alter 'tablex',SPLIT_POLICY => > 'org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy' > However, It gave the information like this and I confused > Unknown argument ignored: SPLIT_POLICY > Updating all regions with the new schema... > So I check the source code That admin.rb might miss the setting for this > argument . > htd.setMaxFileSize(JLong.valueOf(arg.delete(MAX_FILESIZE))) if > arg[MAX_FILESIZE] > htd.setReadOnly(JBoolean.valueOf(arg.delete(READONLY))) if arg[READONLY] > ... > So I think it may be a bug ,is it? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19430) Remove the SettableTimestamp and SettableSequenceId
[ https://issues.apache.org/jira/browse/HBASE-19430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281217#comment-16281217 ] Chia-Ping Tsai commented on HBASE-19430: working at resolving the conflict for branch-2 > Remove the SettableTimestamp and SettableSequenceId > --- > > Key: HBASE-19430 > URL: https://issues.apache.org/jira/browse/HBASE-19430 > Project: HBase > Issue Type: Task >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19430.v0.patch, HBASE-19430.v0.test.patch, > HBASE-19430.v1.patch, HBASE-19430.v1.patch > > > They are introduced by HBASE-11777 and HBASE-12082. Both of them are IA.LP > and denoted with deprecated. To my way of thinking, we should remove them > before 2.0 release. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chance Li updated HBASE-19389: -- Attachment: HBASE-19389-branch-2-V4.patch > Limit concurrency of put with dense (hundreds) columns to prevent write > handler exhausted > - > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0 > > Attachments: CSLM-concurrent-write.png, > HBASE-19389-branch-2-V2.patch, HBASE-19389-branch-2-V3.patch, > HBASE-19389-branch-2-V4.patch, HBASE-19389-branch-2.patch, metrics-1.png, > ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19389) Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
[ https://issues.apache.org/jira/browse/HBASE-19389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281215#comment-16281215 ] Chance Li commented on HBASE-19389: --- patch v4 for checkstyle > Limit concurrency of put with dense (hundreds) columns to prevent write > handler exhausted > - > > Key: HBASE-19389 > URL: https://issues.apache.org/jira/browse/HBASE-19389 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 2.0.0 > Environment: 2000+ Region Servers > PCI-E ssd >Reporter: Chance Li >Assignee: Chance Li > Fix For: 2.0.0 > > Attachments: CSLM-concurrent-write.png, > HBASE-19389-branch-2-V2.patch, HBASE-19389-branch-2-V3.patch, > HBASE-19389-branch-2-V4.patch, HBASE-19389-branch-2.patch, metrics-1.png, > ycsb-result.png > > > In a large cluster, with a large number of clients, we found the RS's > handlers are all busy sometimes. And after investigation we found the root > cause is about CSLM, such as compare function heavy load. We reviewed the > related WALs, and found that there were many columns (more than 1000 columns) > were writing at that time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19301) Provide way for CPs to create short circuited connection with custom configurations
[ https://issues.apache.org/jira/browse/HBASE-19301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281213#comment-16281213 ] stack commented on HBASE-19301: --- .001 subsumes [~anoop.hbase] addendum beefing up the warning and dangers making use of new #getConnection. > Provide way for CPs to create short circuited connection with custom > configurations > --- > > Key: HBASE-19301 > URL: https://issues.apache.org/jira/browse/HBASE-19301 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19301-addendum.patch, > HBASE-19301.master.001.patch, HBASE-19301.patch, HBASE-19301_V2.patch, > HBASE-19301_V2.patch > > > Over in HBASE-18359 we have discussions for this. > Right now HBase provide getConnection() in RegionCPEnv, MasterCPEnv etc. But > this returns a pre created connection (per server). This uses the configs at > hbase-site.xml at that server. > Phoenix needs creating connection in CP with some custom configs. Having this > custom changes in hbase-site.xml is harmful as that will affect all > connections been created at that server. > This issue is for providing an overloaded getConnection(Configuration) API -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19301) Provide way for CPs to create short circuited connection with custom configurations
[ https://issues.apache.org/jira/browse/HBASE-19301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19301: -- Attachment: HBASE-19301.master.001.patch > Provide way for CPs to create short circuited connection with custom > configurations > --- > > Key: HBASE-19301 > URL: https://issues.apache.org/jira/browse/HBASE-19301 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19301-addendum.patch, > HBASE-19301.master.001.patch, HBASE-19301.patch, HBASE-19301_V2.patch, > HBASE-19301_V2.patch > > > Over in HBASE-18359 we have discussions for this. > Right now HBase provide getConnection() in RegionCPEnv, MasterCPEnv etc. But > this returns a pre created connection (per server). This uses the configs at > hbase-site.xml at that server. > Phoenix needs creating connection in CP with some custom configs. Having this > custom changes in hbase-site.xml is harmful as that will affect all > connections been created at that server. > This issue is for providing an overloaded getConnection(Configuration) API -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HBASE-19446) Misspelled 'default' in SimpleRpcScheduler
[
https://issues.apache.org/jira/browse/HBASE-19446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell resolved HBASE-19446.
Resolution: Fixed
Pushed trivial change to branch-1 and branch-1.4. Seems to have been fixed in
later code.
To https://git-wip-us.apache.org/repos/asf/hbase.git
70c7688271..5398d83bc7 branch-1 -> branch-1
099de9088b..8731d9e3e8 branch-1.4 -> branch-1.4
> Misspelled 'default' in SimpleRpcScheduler
> --
>
> Key: HBASE-19446
> URL: https://issues.apache.org/jira/browse/HBASE-19446
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Trivial
> Fix For: 1.4.0
>
>
> Found "deafult' in logs, should be "default" of course. Looks bad, going to
> fix.
> $ find . -name '*.java' -exec fgrep -Hn deafult {} \;
> ./hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java:85:
> callExecutor = new RWQueueRpcExecutor("deafult.RWQ", Math.max(2,
> handlerCount),
> ./hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java:89:
> callExecutor = new FastPathBalancedQueueRpcExecutor("deafult.FPBQ",
> handlerCount,
> ./hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java:92:
> callExecutor = new BalancedQueueRpcExecutor("deafult.BQ",
> handlerCount, maxQueueLength,
> ./hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestSimpleRpcScheduler.java:413:
> envEdge.threadNamePrefixs.add("RpcServer.deafult.FPBQ.Codel.handler");
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19447) INFO level logging of GetClusterStatus from HMaster is too chatty
[ https://issues.apache.org/jira/browse/HBASE-19447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-19447: --- Fix Version/s: 2.0.0 > INFO level logging of GetClusterStatus from HMaster is too chatty > - > > Key: HBASE-19447 > URL: https://issues.apache.org/jira/browse/HBASE-19447 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0 > > > Should be DEBUG. Should be wrapped with a logLevel check. > ./hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java:2409: > LOG.info(getClientIdAuditPrefix() + " get ClusterStatus, status=" + > status); -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HBASE-19447) INFO level logging of GetClusterStatus from HMaster is too chatty
[ https://issues.apache.org/jira/browse/HBASE-19447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-19447. Resolution: Fixed Pushed trivial change To https://git-wip-us.apache.org/repos/asf/hbase.git f88304d162..70c7688271 branch-1 -> branch-1 3bc2a67315..099de9088b branch-1.4 -> branch-1.4 d29ffd4e29..e7a2e4352c branch-2 -> branch-2 4a2e8b852d..97d17ae195 master -> master > INFO level logging of GetClusterStatus from HMaster is too chatty > - > > Key: HBASE-19447 > URL: https://issues.apache.org/jira/browse/HBASE-19447 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 1.4.0 > > > Should be DEBUG. Should be wrapped with a logLevel check. > ./hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java:2409: > LOG.info(getClientIdAuditPrefix() + " get ClusterStatus, status=" + > status); -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19447) INFO level logging of GetClusterStatus from HMaster is too chatty
[ https://issues.apache.org/jira/browse/HBASE-19447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-19447: --- Summary: INFO level logging of GetClusterStatus from HMaster is too chatty (was: INFO level logging of GetClusterStatus is too chatty) > INFO level logging of GetClusterStatus from HMaster is too chatty > - > > Key: HBASE-19447 > URL: https://issues.apache.org/jira/browse/HBASE-19447 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 1.4.0 > > > Should be DEBUG. Should be wrapped with a logLevel check. > ./hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java:2409: > LOG.info(getClientIdAuditPrefix() + " get ClusterStatus, status=" + > status); -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19447) INFO level logging of GetClusterStatus is too chatty
[ https://issues.apache.org/jira/browse/HBASE-19447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281201#comment-16281201 ] Andrew Purtell commented on HBASE-19447: We don't do this kind of logging elsewhere outside of TRACE level logging in RPC, so just remove it. > INFO level logging of GetClusterStatus is too chatty > > > Key: HBASE-19447 > URL: https://issues.apache.org/jira/browse/HBASE-19447 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 1.4.0 > > > Should be DEBUG. Should be wrapped with a logLevel check. > ./hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java:2409: > LOG.info(getClientIdAuditPrefix() + " get ClusterStatus, status=" + > status); -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19447) INFO level logging of GetClusterStatus is too chatty
Andrew Purtell created HBASE-19447: -- Summary: INFO level logging of GetClusterStatus is too chatty Key: HBASE-19447 URL: https://issues.apache.org/jira/browse/HBASE-19447 Project: HBase Issue Type: Bug Affects Versions: 1.4.0 Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 1.4.0 Should be DEBUG. Should be wrapped with a logLevel check. ./hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java:2409: LOG.info(getClientIdAuditPrefix() + " get ClusterStatus, status=" + status); -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19446) Misspelled 'default' in SimpleRpcScheduler
Andrew Purtell created HBASE-19446:
--
Summary: Misspelled 'default' in SimpleRpcScheduler
Key: HBASE-19446
URL: https://issues.apache.org/jira/browse/HBASE-19446
Project: HBase
Issue Type: Bug
Affects Versions: 1.4.0
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Trivial
Fix For: 1.4.0
Found "deafult' in logs, should be "default" of course. Looks bad, going to
fix.
$ find . -name '*.java' -exec fgrep -Hn deafult {} \;
./hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java:85:
callExecutor = new RWQueueRpcExecutor("deafult.RWQ", Math.max(2,
handlerCount),
./hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java:89:
callExecutor = new FastPathBalancedQueueRpcExecutor("deafult.FPBQ",
handlerCount,
./hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java:92:
callExecutor = new BalancedQueueRpcExecutor("deafult.BQ", handlerCount,
maxQueueLength,
./hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestSimpleRpcScheduler.java:413:
envEdge.threadNamePrefixs.add("RpcServer.deafult.FPBQ.Codel.handler");
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19435) Reopen Files for ClosedChannelException in BucketCache
[ https://issues.apache.org/jira/browse/HBASE-19435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-19435: -- Attachment: HBASE-19435.master.005.patch > Reopen Files for ClosedChannelException in BucketCache > -- > > Key: HBASE-19435 > URL: https://issues.apache.org/jira/browse/HBASE-19435 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.0.0, 1.3.1 >Reporter: Zach York >Assignee: Zach York > Attachments: HBASE-19435.master.001.patch, > HBASE-19435.master.002.patch, HBASE-19435.master.003.patch, > HBASE-19435.master.004.patch, HBASE-19435.master.005.patch > > > When using the FileIOEngine for BucketCache, the cache will be disabled if > the connection is interrupted or closed. HBase will then get > ClosedChannelExceptions trying to access the file. After 60s, the RS will > disable the cache. This causes severe read performance degradation for > workloads that rely on this cache. FileIOEngine never tries to reopen the > connection. This JIRA is to reopen files when the BucketCache encounters a > ClosedChannelException. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-15482) Provide an option to skip calculating block locations for SnapshotInputFormat
[ https://issues.apache.org/jira/browse/HBASE-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiang Li updated HBASE-15482: - Fix Version/s: 2.1.0 Status: Patch Available (was: Open) > Provide an option to skip calculating block locations for SnapshotInputFormat > - > > Key: HBASE-15482 > URL: https://issues.apache.org/jira/browse/HBASE-15482 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: Liyin Tang >Assignee: Xiang Li >Priority: Minor > Fix For: 2.1.0 > > Attachments: HBASE-15482.master.000.patch > > > When a MR job is reading from SnapshotInputFormat, it needs to calculate the > splits based on the block locations in order to get best locality. However, > this process may take a long time for large snapshots. > In some setup, the computing layer, Spark, Hive or Presto could run out side > of HBase cluster. In these scenarios, the block locality doesn't matter. > Therefore, it will be great to have an option to skip calculating the block > locations for every job. That will super useful for the Hive/Presto/Spark > connectors. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19445) PerformanceEvaluation NPE processing split policy option
[ https://issues.apache.org/jira/browse/HBASE-19445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281197#comment-16281197 ] Hudson commented on HBASE-19445: FAILURE: Integrated in Jenkins build HBase-1.4 #1053 (See [https://builds.apache.org/job/HBase-1.4/1053/]) HBASE-19445 PerformanceEvaluation NPE processing split policy option (apurtell: rev 6fcbdc0cc799ae8df965b4a0677e6fe2f1f7a54c) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java > PerformanceEvaluation NPE processing split policy option > > > Key: HBASE-19445 > URL: https://issues.apache.org/jira/browse/HBASE-19445 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Trivial > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-19445-branch-1.patch, HBASE-19445.patch > > > $ ./bin/hbase pe --nomapred randomWrite 10 > java.lang.NullPointerException > at > org.apache.hadoop.hbase.PerformanceEvaluation.getTableDescriptor(PerformanceEvaluation.java:387) > at > org.apache.hadoop.hbase.PerformanceEvaluation.checkTable(PerformanceEvaluation.java:356) > at > org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:1778) > at > org.apache.hadoop.hbase.PerformanceEvaluation.run(PerformanceEvaluation.java:2168) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at > org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:2189) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19435) Reopen Files for ClosedChannelException in BucketCache
[ https://issues.apache.org/jira/browse/HBASE-19435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-19435: -- Status: In Progress (was: Patch Available) > Reopen Files for ClosedChannelException in BucketCache > -- > > Key: HBASE-19435 > URL: https://issues.apache.org/jira/browse/HBASE-19435 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 1.3.1, 2.0.0 >Reporter: Zach York >Assignee: Zach York > Attachments: HBASE-19435.master.001.patch, > HBASE-19435.master.002.patch, HBASE-19435.master.003.patch, > HBASE-19435.master.004.patch > > > When using the FileIOEngine for BucketCache, the cache will be disabled if > the connection is interrupted or closed. HBase will then get > ClosedChannelExceptions trying to access the file. After 60s, the RS will > disable the cache. This causes severe read performance degradation for > workloads that rely on this cache. FileIOEngine never tries to reopen the > connection. This JIRA is to reopen files when the BucketCache encounters a > ClosedChannelException. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19435) Reopen Files for ClosedChannelException in BucketCache
[ https://issues.apache.org/jira/browse/HBASE-19435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-19435: -- Status: Patch Available (was: In Progress) > Reopen Files for ClosedChannelException in BucketCache > -- > > Key: HBASE-19435 > URL: https://issues.apache.org/jira/browse/HBASE-19435 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 1.3.1, 2.0.0 >Reporter: Zach York >Assignee: Zach York > Attachments: HBASE-19435.master.001.patch, > HBASE-19435.master.002.patch, HBASE-19435.master.003.patch, > HBASE-19435.master.004.patch > > > When using the FileIOEngine for BucketCache, the cache will be disabled if > the connection is interrupted or closed. HBase will then get > ClosedChannelExceptions trying to access the file. After 60s, the RS will > disable the cache. This causes severe read performance degradation for > workloads that rely on this cache. FileIOEngine never tries to reopen the > connection. This JIRA is to reopen files when the BucketCache encounters a > ClosedChannelException. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19435) Reopen Files for ClosedChannelException in BucketCache
[ https://issues.apache.org/jira/browse/HBASE-19435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-19435: -- Attachment: HBASE-19435.master.004.patch > Reopen Files for ClosedChannelException in BucketCache > -- > > Key: HBASE-19435 > URL: https://issues.apache.org/jira/browse/HBASE-19435 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.0.0, 1.3.1 >Reporter: Zach York >Assignee: Zach York > Attachments: HBASE-19435.master.001.patch, > HBASE-19435.master.002.patch, HBASE-19435.master.003.patch, > HBASE-19435.master.004.patch > > > When using the FileIOEngine for BucketCache, the cache will be disabled if > the connection is interrupted or closed. HBase will then get > ClosedChannelExceptions trying to access the file. After 60s, the RS will > disable the cache. This causes severe read performance degradation for > workloads that rely on this cache. FileIOEngine never tries to reopen the > connection. This JIRA is to reopen files when the BucketCache encounters a > ClosedChannelException. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19435) Reopen Files for ClosedChannelException in BucketCache
[ https://issues.apache.org/jira/browse/HBASE-19435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-19435: -- Attachment: HBASE-19435.master.003.patch > Reopen Files for ClosedChannelException in BucketCache > -- > > Key: HBASE-19435 > URL: https://issues.apache.org/jira/browse/HBASE-19435 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.0.0, 1.3.1 >Reporter: Zach York >Assignee: Zach York > Attachments: HBASE-19435.master.001.patch, > HBASE-19435.master.002.patch, HBASE-19435.master.003.patch > > > When using the FileIOEngine for BucketCache, the cache will be disabled if > the connection is interrupted or closed. HBase will then get > ClosedChannelExceptions trying to access the file. After 60s, the RS will > disable the cache. This causes severe read performance degradation for > workloads that rely on this cache. FileIOEngine never tries to reopen the > connection. This JIRA is to reopen files when the BucketCache encounters a > ClosedChannelException. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-15628) Implement an AsyncOutputStream which can work with any FileSystem implementation
[ https://issues.apache.org/jira/browse/HBASE-15628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281190#comment-16281190 ] Hudson commented on HBASE-15628: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4181 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4181/]) HBASE-15628 Implement an AsyncOutputStream which can work with any (stack: rev 00750fe79acbb6a43daa62b9fcabbd1f5ce4cf6c) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/asyncfs/FanOutOneBlockAsyncDFSOutputHelper.java > Implement an AsyncOutputStream which can work with any FileSystem > implementation > > > Key: HBASE-15628 > URL: https://issues.apache.org/jira/browse/HBASE-15628 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-15628-v1.patch, HBASE-15628-v2.patch, > HBASE-15628-v3.patch, HBASE-15628.patch > > > Used for testcase only. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19430) Remove the SettableTimestamp and SettableSequenceId
[ https://issues.apache.org/jira/browse/HBASE-19430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281184#comment-16281184 ] Chia-Ping Tsai commented on HBASE-19430: Will commit the patch with checkstyle fix > Remove the SettableTimestamp and SettableSequenceId > --- > > Key: HBASE-19430 > URL: https://issues.apache.org/jira/browse/HBASE-19430 > Project: HBase > Issue Type: Task >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19430.v0.patch, HBASE-19430.v0.test.patch, > HBASE-19430.v1.patch, HBASE-19430.v1.patch > > > They are introduced by HBASE-11777 and HBASE-12082. Both of them are IA.LP > and denoted with deprecated. To my way of thinking, we should remove them > before 2.0 release. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-19433) ChangeSplitPolicyAction modifies an immutable HTableDescriptor
[
https://issues.apache.org/jira/browse/HBASE-19433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281168#comment-16281168
]
Ted Yu commented on HBASE-19433:
{code}
Mockito.when((HBaseTestingUtility)ctx.getHBaseIntegrationTestingUtility()).thenReturn((HBaseTestingUtility)TEST_UTIL);
{code}
leads to:
{code}
org.mockito.exceptions.misusing.WrongTypeOfReturnValue:
HBaseTestingUtility cannot be returned by getHBaseIntegrationTestingUtility()
getHBaseIntegrationTestingUtility() should return IntegrationTestingUtility
{code}
> ChangeSplitPolicyAction modifies an immutable HTableDescriptor
> --
>
> Key: HBASE-19433
> URL: https://issues.apache.org/jira/browse/HBASE-19433
> Project: HBase
> Issue Type: Bug
> Components: integration tests
>Reporter: Josh Elser
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: 19433-unsupported.out, 19433.v1.txt, 19433.v2.txt,
> 19433.v3.txt, 19433.v4.txt
>
>
> {noformat}
> 2017-12-01 23:18:51,433 WARN [ChaosMonkeyThread] policies.Policy: Exception
> occurred during performing action: java.lang.UnsupportedOperationException:
> HTableDescriptor is read-only
> at
> org.apache.hadoop.hbase.client.ImmutableHTableDescriptor.getDelegateeForModification(ImmutableHTableDescriptor.java:59)
> at
> org.apache.hadoop.hbase.HTableDescriptor.setRegionSplitPolicyClassName(HTableDescriptor.java:333)
> at
> org.apache.hadoop.hbase.chaos.actions.ChangeSplitPolicyAction.perform(ChangeSplitPolicyAction.java:54)
> at
> org.apache.hadoop.hbase.chaos.policies.PeriodicRandomActionPolicy.runOneIteration(PeriodicRandomActionPolicy.java:59)
> at
> org.apache.hadoop.hbase.chaos.policies.PeriodicPolicy.run(PeriodicPolicy.java:41)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Found during some internal testing. Need to make sure this Action, in
> addition to the other, don't fall into the trap of modifying the
> TableDescriptor obtained from Admin.
> [~tedyu], want to take a stab at it?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-19433) ChangeSplitPolicyAction modifies an immutable HTableDescriptor
[
https://issues.apache.org/jira/browse/HBASE-19433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281163#comment-16281163
]
Josh Elser commented on HBASE-19433:
bq. The above change is for passing mockito type checking.
I assumed as much, but it seems silly to me to add a new method to return the
same object when we can work around it in the test with a cast. Am I missing
something?
> ChangeSplitPolicyAction modifies an immutable HTableDescriptor
> --
>
> Key: HBASE-19433
> URL: https://issues.apache.org/jira/browse/HBASE-19433
> Project: HBase
> Issue Type: Bug
> Components: integration tests
>Reporter: Josh Elser
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0-beta-1
>
> Attachments: 19433-unsupported.out, 19433.v1.txt, 19433.v2.txt,
> 19433.v3.txt, 19433.v4.txt
>
>
> {noformat}
> 2017-12-01 23:18:51,433 WARN [ChaosMonkeyThread] policies.Policy: Exception
> occurred during performing action: java.lang.UnsupportedOperationException:
> HTableDescriptor is read-only
> at
> org.apache.hadoop.hbase.client.ImmutableHTableDescriptor.getDelegateeForModification(ImmutableHTableDescriptor.java:59)
> at
> org.apache.hadoop.hbase.HTableDescriptor.setRegionSplitPolicyClassName(HTableDescriptor.java:333)
> at
> org.apache.hadoop.hbase.chaos.actions.ChangeSplitPolicyAction.perform(ChangeSplitPolicyAction.java:54)
> at
> org.apache.hadoop.hbase.chaos.policies.PeriodicRandomActionPolicy.runOneIteration(PeriodicRandomActionPolicy.java:59)
> at
> org.apache.hadoop.hbase.chaos.policies.PeriodicPolicy.run(PeriodicPolicy.java:41)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Found during some internal testing. Need to make sure this Action, in
> addition to the other, don't fall into the trap of modifying the
> TableDescriptor obtained from Admin.
> [~tedyu], want to take a stab at it?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
