[jira] [Created] (HBASE-19622) Reimplement ReplicationPeers with the new replication storage interface
Guanghao Zhang created HBASE-19622: -- Summary: Reimplement ReplicationPeers with the new replication storage interface Key: HBASE-19622 URL: https://issues.apache.org/jira/browse/HBASE-19622 Project: HBase Issue Type: Bug Components: proc-v2, Replication Reporter: Guanghao Zhang Fix For: HBASE-19397 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
Re: Accessing region state via web UI
REST endpoint more natural loci for programmatic consumers imo On Dec 24, 2017 5:22 PM, "Apekshit Sharma" wrote: > I started looking into more after last reply. > I think exposing this info can be useful to the operators - dashboard of > region states, or tools to quickly dump list of regions which require care, > etc > But wasn't thinking about hbase rest api so far. > I think web ui is not the right endpoint to expose json api when we already > have a whole well designed way to do it - REST apis. > It should rather be an addition to Table endpoints (table 14 here: > http://hbase.apache.org/book.html#_using_rest_endpoints). > > -- Appy > > On Sun, Dec 24, 2017 at 5:16 PM, Apekshit Sharma > wrote: > > > More details: > > Proposed interface is json. > > Pros: > > - more info exposed to operators, can be used for scripting > > > > Cons: > > - We don't have functions in Admin to expose region states > > - We don't expose much programatic info through our web ui. > > > > -- Appy > > > > > > > > On Fri, Dec 22, 2017 at 6:01 PM, Ted Yu wrote: > > > >> Hi, > >> See (tail of) HBASE-17436 for detailed background information. > >> > >> Here is the summary: > >> > >> In HBASE-17436 (and HBASE-19594), region state is accessed thru web UI. > >> This is for helping operators gain better insight into the cluster > status. > >> > >> However, since RegionState is marked @InterfaceAudience.Private , we > (Appy > >> and I) want to present this discussion to the community to raise > >> awareness. > >> > >> Your comment is welcome. > >> > > > > > > > > -- > > > > -- Appy > > > > > > -- > > -- Appy >
[jira] [Created] (HBASE-19621) Revisit the methods in ReplicationPeerConfigBuilder
Guanghao Zhang created HBASE-19621: -- Summary: Revisit the methods in ReplicationPeerConfigBuilder Key: HBASE-19621 URL: https://issues.apache.org/jira/browse/HBASE-19621 Project: HBase Issue Type: Bug Reporter: Guanghao Zhang Assignee: Guanghao Zhang Add 4 methods for ReplicationPeerConfigBuilder: addConfiguration addAllConfiguration addPeerData addAllPeerData Meanwhile, remove setConfiuration and serPeerData from ReplicationPeerConfigBuilder. Because previous ReplicationPeerConfig didn't support setConfiuration and serPeerData. And previous code used getConfiguration.put or putAll to add configuration. So add methods to keep consistent with old usage. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
Re: [jira] [Resolved] (HBASE-15453) [Performance] Considering reverting HBASE-10015 - reinstate synchronized in StoreScanner
On Dec 24, 2017 00:05, "Lars Hofhansl (JIRA)" wrote: > > [ https://issues.apache.org/jira/browse/HBASE-15453?page= > com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] > > Lars Hofhansl resolved HBASE-15453. > --- > Resolution: Won't Fix > > Lemme just close. In 1.3+ it's not an issue anyway (the need to > synchronize is gone there) > > > [Performance] Considering reverting HBASE-10015 - reinstate synchronized > in StoreScanner > > > > > > > Key: HBASE-15453 > > URL: https://issues.apache.org/jira/browse/HBASE-15453 > > Project: HBase > > Issue Type: Improvement > > Components: Performance > >Reporter: Lars Hofhansl > >Assignee: Lars Hofhansl > >Priority: Critical > > Attachments: 15453-0.98.txt > > > > > > In HBASE-10015 back then I found that intrinsic locks (synchronized) in > StoreScanner are slower that explicit locks. > > I was surprised by this. To make sure I added a simple perf test and > many folks ran it on their machines. All found that explicit locks were > faster. > > Now... I just ran that test again. On the latest JDK8 I find that now > the intrinsic locks are significantly faster: > > (OpenJDK Runtime Environment (build 1.8.0_72-b15)) > > Explicit locks: > > 10 runs mean:2223.6 sigma:72.29412147609237 > > Intrinsic locks: > > 10 runs mean:1865.3 sigma:32.63755505548784 > > I confirmed the same with timing some Phoenix scans. We can save a bunch > of time by changing this back > > Arrghhh... So maybe it's time to revert this now...? > > (Note that in trunk due to [~ram_krish]'s work, we do not lock in > StoreScanner anymore) > > I'll attach the perf test and a patch that changes lock to synchronized, > if some folks could run this on 0.98, that'd be great. > > > > -- > This message was sent by Atlassian JIRA > (v6.4.14#64029) >
Re: [jira] [Resolved] (HBASE-15453) [Performance] Considering reverting HBASE-10015 - reinstate synchronized in StoreScanner
User On Dec 24, 2017 00:05, "Lars Hofhansl (JIRA)" wrote: [ https://issues.apache.org/jira/browse/HBASE-15453?page= com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-15453. --- Resolution: Won't Fix Lemme just close. In 1.3+ it's not an issue anyway (the need to synchronize is gone there) > [Performance] Considering reverting HBASE-10015 - reinstate synchronized in StoreScanner > > > Key: HBASE-15453 > URL: https://issues.apache.org/jira/browse/HBASE-15453 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Attachments: 15453-0.98.txt > > > In HBASE-10015 back then I found that intrinsic locks (synchronized) in StoreScanner are slower that explicit locks. > I was surprised by this. To make sure I added a simple perf test and many folks ran it on their machines. All found that explicit locks were faster. > Now... I just ran that test again. On the latest JDK8 I find that now the intrinsic locks are significantly faster: > (OpenJDK Runtime Environment (build 1.8.0_72-b15)) > Explicit locks: > 10 runs mean:2223.6 sigma:72.29412147609237 > Intrinsic locks: > 10 runs mean:1865.3 sigma:32.63755505548784 > I confirmed the same with timing some Phoenix scans. We can save a bunch of time by changing this back > Arrghhh... So maybe it's time to revert this now...? > (Note that in trunk due to [~ram_krish]'s work, we do not lock in StoreScanner anymore) > I'll attach the perf test and a patch that changes lock to synchronized, if some folks could run this on 0.98, that'd be great. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19620) Add UT to confirm the race in MasterRpcServices.getProcedureResult
Duo Zhang created HBASE-19620: - Summary: Add UT to confirm the race in MasterRpcServices.getProcedureResult Key: HBASE-19620 URL: https://issues.apache.org/jira/browse/HBASE-19620 Project: HBase Issue Type: Bug Components: proc-v2, test Reporter: Duo Zhang -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19619) TestReplicationShell fails due to calling Collections$UnmodifiableCollection.add
Ted Yu created HBASE-19619: -- Summary: TestReplicationShell fails due to calling Collections$UnmodifiableCollection.add Key: HBASE-19619 URL: https://issues.apache.org/jira/browse/HBASE-19619 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Ted Yu Here is the error: {code} Error: test_append_peer_namespaces:_works_with_namespaces_array(Hbase::ReplicationAdminTest): Java::JavaLang::UnsupportedOperationException: java.util.Collections$UnmodifiableCollection.add(java/util/Collections.java:1055) java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498) org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:453) org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:314) RUBY.block in add_peer_namespaces(/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-shell/src/main/ruby/hbase/replication_admin.rb:226) org.jruby.RubyArray.each(org/jruby/RubyArray.java:1734) org.jruby.RubyArray$INVOKER$i$0$0$each.call(org/jruby/RubyArray$INVOKER$i$0$0$each.gen) RUBY.add_peer_namespaces(/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-shell/src/main/ruby/hbase/replication_admin.rb:225) RUBY.command(/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-shell/src/main/ruby/shell/commands/append_peer_namespaces.rb:40) org.jruby.RubyClass.finvoke(org/jruby/RubyClass.java:522) {code} The cause was due to adding to UnmodifiableCollection -- This message was sent by Atlassian JIRA (v6.4.14#64029)
Re: Accessing region state via web UI
See this comment for the suggestion of exposing thru JSON: https://issues.apache.org/jira/browse/HBASE-17436?focusedCommentId=16007568&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16007568 FYI On Sun, Dec 24, 2017 at 5:22 PM, Apekshit Sharma wrote: > I started looking into more after last reply. > I think exposing this info can be useful to the operators - dashboard of > region states, or tools to quickly dump list of regions which require care, > etc > But wasn't thinking about hbase rest api so far. > I think web ui is not the right endpoint to expose json api when we already > have a whole well designed way to do it - REST apis. > It should rather be an addition to Table endpoints (table 14 here: > http://hbase.apache.org/book.html#_using_rest_endpoints). > > -- Appy > > On Sun, Dec 24, 2017 at 5:16 PM, Apekshit Sharma > wrote: > > > More details: > > Proposed interface is json. > > Pros: > > - more info exposed to operators, can be used for scripting > > > > Cons: > > - We don't have functions in Admin to expose region states > > - We don't expose much programatic info through our web ui. > > > > -- Appy > > > > > > > > On Fri, Dec 22, 2017 at 6:01 PM, Ted Yu wrote: > > > >> Hi, > >> See (tail of) HBASE-17436 for detailed background information. > >> > >> Here is the summary: > >> > >> In HBASE-17436 (and HBASE-19594), region state is accessed thru web UI. > >> This is for helping operators gain better insight into the cluster > status. > >> > >> However, since RegionState is marked @InterfaceAudience.Private , we > (Appy > >> and I) want to present this discussion to the community to raise > >> awareness. > >> > >> Your comment is welcome. > >> > > > > > > > > -- > > > > -- Appy > > > > > > -- > > -- Appy >
[jira] [Created] (HBASE-19618) Remove replicationQueuesClient.class/replicationQueues.class config from ReplicationFactory
Guanghao Zhang created HBASE-19618: -- Summary: Remove replicationQueuesClient.class/replicationQueues.class config from ReplicationFactory Key: HBASE-19618 URL: https://issues.apache.org/jira/browse/HBASE-19618 Project: HBase Issue Type: Bug Reporter: Guanghao Zhang When implement the procedure of replication admin operations, we abstract a replication storage interface in HBASE-19543. So ReplicationQueues/ReplicationQueuesClient are not used anymore. These interface are IA.private. So it is ok to remove them. But there are two config: hbase.region.replica.replication.replicationQueues.class and hbase.region.replica.replication.replicationQueuesClient.class in ReplicationFactory. These configs were introduced by HBASE-15867, which only in 2.0. And the feature development is not active now. In the future, we can implement the table based replication to replication storage interface. So let's remove them before release 2.0. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
Re: Accessing region state via web UI
I started looking into more after last reply. I think exposing this info can be useful to the operators - dashboard of region states, or tools to quickly dump list of regions which require care, etc But wasn't thinking about hbase rest api so far. I think web ui is not the right endpoint to expose json api when we already have a whole well designed way to do it - REST apis. It should rather be an addition to Table endpoints (table 14 here: http://hbase.apache.org/book.html#_using_rest_endpoints). -- Appy On Sun, Dec 24, 2017 at 5:16 PM, Apekshit Sharma wrote: > More details: > Proposed interface is json. > Pros: > - more info exposed to operators, can be used for scripting > > Cons: > - We don't have functions in Admin to expose region states > - We don't expose much programatic info through our web ui. > > -- Appy > > > > On Fri, Dec 22, 2017 at 6:01 PM, Ted Yu wrote: > >> Hi, >> See (tail of) HBASE-17436 for detailed background information. >> >> Here is the summary: >> >> In HBASE-17436 (and HBASE-19594), region state is accessed thru web UI. >> This is for helping operators gain better insight into the cluster status. >> >> However, since RegionState is marked @InterfaceAudience.Private , we (Appy >> and I) want to present this discussion to the community to raise >> awareness. >> >> Your comment is welcome. >> > > > > -- > > -- Appy > -- -- Appy
Re: Accessing region state via web UI
More details: Proposed interface is json. Pros: - more info exposed to operators, can be used for scripting Cons: - We don't have functions in Admin to expose region states - We don't expose much programatic info through our web ui. -- Appy On Fri, Dec 22, 2017 at 6:01 PM, Ted Yu wrote: > Hi, > See (tail of) HBASE-17436 for detailed background information. > > Here is the summary: > > In HBASE-17436 (and HBASE-19594), region state is accessed thru web UI. > This is for helping operators gain better insight into the cluster status. > > However, since RegionState is marked @InterfaceAudience.Private , we (Appy > and I) want to present this discussion to the community to raise awareness. > > Your comment is welcome. > -- -- Appy
[jira] [Created] (HBASE-19617) Remove ReplicationQueues, use ReplicationQueueStorage directly
Duo Zhang created HBASE-19617: - Summary: Remove ReplicationQueues, use ReplicationQueueStorage directly Key: HBASE-19617 URL: https://issues.apache.org/jira/browse/HBASE-19617 Project: HBase Issue Type: Sub-task Components: Replication Reporter: Duo Zhang -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19616) Review of LogCleaner Class
BELUGA BEHR created HBASE-19616: --- Summary: Review of LogCleaner Class Key: HBASE-19616 URL: https://issues.apache.org/jira/browse/HBASE-19616 Project: HBase Issue Type: Improvement Components: hbase Affects Versions: 3.0.0 Reporter: BELUGA BEHR Priority: Trivial * Parameterize logging * Remove compiler-reported dead code to re-enabling useful logging * Use ArrayList instead of LinkedList -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19615) CompositeImmutableSegment ArrayList Instead of LinkedList
BELUGA BEHR created HBASE-19615: --- Summary: CompositeImmutableSegment ArrayList Instead of LinkedList Key: HBASE-19615 URL: https://issues.apache.org/jira/browse/HBASE-19615 Project: HBase Issue Type: Improvement Components: hbase Affects Versions: 3.0.0 Reporter: BELUGA BEHR Priority: Trivial CompositeImmutableSegment ArrayList Instead of LinkedList The code is essentially making a copy of an existing List object that doesn't seem to be modified, it is only being iterated over. No need to build an entire LinkedList structure for that. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19614) CompoundBloomFilterWriter Array Instead of LinkedList
BELUGA BEHR created HBASE-19614: --- Summary: CompoundBloomFilterWriter Array Instead of LinkedList Key: HBASE-19614 URL: https://issues.apache.org/jira/browse/HBASE-19614 Project: HBase Issue Type: Improvement Components: hbase Affects Versions: 3.0.0 Reporter: BELUGA BEHR Priority: Trivial {quote} Resizable-array implementation of the Deque interface. Array deques have no capacity restrictions; they grow as necessary to support usage. They are not thread-safe; in the absence of external synchronization, they do not support concurrent access by multiple threads. Null elements are prohibited. This class *is likely to be faster than* Stack when used as a stack, and faster than *LinkedList when used as a queue.* {quote} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19613) Review of WALSplitter Class
BELUGA BEHR created HBASE-19613: --- Summary: Review of WALSplitter Class Key: HBASE-19613 URL: https://issues.apache.org/jira/browse/HBASE-19613 Project: HBase Issue Type: Improvement Components: hbase Affects Versions: 3.0.0 Reporter: BELUGA BEHR Priority: Trivial * Use ArrayList instead LinkedList * Use Apache Commons where appropriate * Parameterize and improve logging -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19612) Review of ZKUtil Class
BELUGA BEHR created HBASE-19612: --- Summary: Review of ZKUtil Class Key: HBASE-19612 URL: https://issues.apache.org/jira/browse/HBASE-19612 Project: HBase Issue Type: Improvement Components: hbase Affects Versions: 3.0.0 Reporter: BELUGA BEHR Priority: Trivial * Use Apache Commons where appropriate * Use parameterized logging of SLF4J * Fix Typos in comment * Use Arrays Instead of LinkedLists for performance sake -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19611) Review of QuotaRetriever Class
BELUGA BEHR created HBASE-19611: --- Summary: Review of QuotaRetriever Class Key: HBASE-19611 URL: https://issues.apache.org/jira/browse/HBASE-19611 Project: HBase Issue Type: Improvement Components: hbase Affects Versions: 3.0.0 Reporter: BELUGA BEHR Priority: Trivial * Use ArrayDeque instead of LinkedList for performanc * Remove use of {{StringUtils.stringifyException}} in favor of SLF4J formatting * Remove unused imports -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19610) Fix Checkstyle errors in hbase-protocol
Jan Hentschel created HBASE-19610: - Summary: Fix Checkstyle errors in hbase-protocol Key: HBASE-19610 URL: https://issues.apache.org/jira/browse/HBASE-19610 Project: HBase Issue Type: Sub-task Reporter: Jan Hentschel Assignee: Jan Hentschel Priority: Minor Fix the remaining Checkstyle errors in the *hbase-protocol* module and enable Checkstyle to fail on violations. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19609) Fix Checkstyle errors in hbase-metrics
Jan Hentschel created HBASE-19609: - Summary: Fix Checkstyle errors in hbase-metrics Key: HBASE-19609 URL: https://issues.apache.org/jira/browse/HBASE-19609 Project: HBase Issue Type: Sub-task Components: metrics Reporter: Jan Hentschel Assignee: Jan Hentschel Priority: Minor Fix the remaining Checkstyle errors in the *hbase-metrics* module and enable Checkstyle to fail on violations. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19608) Race in MasterRpcServices.getProcedureResult
Duo Zhang created HBASE-19608: - Summary: Race in MasterRpcServices.getProcedureResult Key: HBASE-19608 URL: https://issues.apache.org/jira/browse/HBASE-19608 Project: HBase Issue Type: Bug Components: proc-v2 Reporter: Duo Zhang Assignee: Duo Zhang Fix For: 2.0.0-beta-1 In ProcedureExecutor, we only consider a procedure to be finished when it is removed from the procedures map. But in MasterRpcServices.getProcedureResult, we will use the state of the procedure to determine whether it is finished. If setFailure is called, the procedure may enter the FAILED state which indicates it is finished before being removed from the procedure map, and then causes an assertion when we calling ProcedureExecutor.removeResult in MasterRpcServices.getProcedureResult. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HBASE-19125) TestReplicator is flaky
[ https://issues.apache.org/jira/browse/HBASE-19125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved HBASE-19125. Resolution: Duplicate > TestReplicator is flaky > --- > > Key: HBASE-19125 > URL: https://issues.apache.org/jira/browse/HBASE-19125 > Project: HBase > Issue Type: Bug >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 2.0.0, 3.0.0, 1.4.1, 1.5.0 > > > TestReplicator fails now and again. I had a look at the test. This is > something I contributed a while back but looking at it again it needs a > different approach. I'm going to disable it for now until this issue is > resolved. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HBASE-19523) TestLogRolling is flakey
[ https://issues.apache.org/jira/browse/HBASE-19523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved HBASE-19523. Resolution: Duplicate see HBASE-19559 > TestLogRolling is flakey > > > Key: HBASE-19523 > URL: https://issues.apache.org/jira/browse/HBASE-19523 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Duo Zhang > > https://builds.apache.org/job/PreCommit-HBASE-Build/10475/testReport/ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-19607) throw the exception if memstore is incompetent to do the deep clone for cell
Chia-Ping Tsai created HBASE-19607: -- Summary: throw the exception if memstore is incompetent to do the deep clone for cell Key: HBASE-19607 URL: https://issues.apache.org/jira/browse/HBASE-19607 Project: HBase Issue Type: Bug Reporter: Chia-Ping Tsai Priority: Trivial We must to clone the cell since the cell is backed by the reusable byte array. Also, we assume all cells passed to AbstractMemStore is of ExtendedCell. Not only is the type check unnecessary, but it also confuses the readers. {code:title=AbstractMemStore.java} private static Cell deepCopyIfNeeded(Cell cell) { if (cell instanceof ExtendedCell) { return ((ExtendedCell) cell).deepClone(); } return cell; } {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)