[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200603#comment-17200603
]
Danil Lipovoy commented on HBASE-23887:
---
[~elserj] could you explain please, how to enable AdaptiveLruBlockCache ?
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png,
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> image-2020-09-23-09-48-59-714.png, ratio.png, ratio2.png,
> read_requests_100pBC_vs_23pBC.png, requests_100p.png, requests_100p.png,
> requests_new2_100p.png, requests_new_100p.png, scan.png, scan_and_gets.png,
> scan_and_gets2.png, wave.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache9 commented on a change in pull request #2440: HBASE-25082: Per table WAL metrics: appendCount and appendSize
Apache9 commented on a change in pull request #2440: URL: https://github.com/apache/hbase/pull/2440#discussion_r493237228 ## File path: hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsWALSource.java ## @@ -114,6 +115,4 @@ void incrementSizeLogRoll(); void incrementWrittenBytes(long val); - - long getWrittenBytes(); Review comment: Why this is removed? It is not used? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200601#comment-17200601
]
Danil Lipovoy commented on HBASE-23887:
---
[~vrodionov], I did simple test - disable BC and run YCSB on table 100 GB.
Without the cache is working very slow:
!image-2020-09-23-09-48-59-714.png!
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png,
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> image-2020-09-23-09-48-59-714.png, ratio.png, ratio2.png,
> read_requests_100pBC_vs_23pBC.png, requests_100p.png, requests_100p.png,
> requests_new2_100p.png, requests_new_100p.png, scan.png, scan_and_gets.png,
> scan_and_gets2.png, wave.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danil Lipovoy updated HBASE-23887:
--
Attachment: image-2020-09-23-09-48-59-714.png
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png,
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> image-2020-09-23-09-48-59-714.png, ratio.png, ratio2.png,
> read_requests_100pBC_vs_23pBC.png, requests_100p.png, requests_100p.png,
> requests_new2_100p.png, requests_new_100p.png, scan.png, scan_and_gets.png,
> scan_and_gets2.png, wave.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to make Pull Request, hope it is right way to make some
> contribution in this cool product.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-25073) Should not use XXXService.Interface.class.getSimpleName as stub key prefix in AsyncConnectionImpl
[ https://issues.apache.org/jira/browse/HBASE-25073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-25073. --- Fix Version/s: 2.2.7 2.4.0 2.3.3 3.0.0-alpha-1 Hadoop Flags: Reviewed Resolution: Fixed Pushed to branch-2.2+. Thanks [~ww112925] for contributing. > Should not use XXXService.Interface.class.getSimpleName as stub key prefix in > AsyncConnectionImpl > - > > Key: HBASE-25073 > URL: https://issues.apache.org/jira/browse/HBASE-25073 > Project: HBase > Issue Type: Task > Components: Client >Reporter: Duo Zhang >Assignee: wangwei >Priority: Major > Labels: beginner, trivial > Fix For: 3.0.0-alpha-1, 2.3.3, 2.4.0, 2.2.7 > > > It will just return "Interface" for all services. This is not a bug for > AsyncConnectionImpl as we have different maps for ClientService and > AdminService, but it is still a bit confusing to developers. > I suggest either we just use empty string and add a comment to say that the > service name is not important here, or change to use > XXXService.class.getSimpleName() or XXXService.getDescriptor().getName(), so > it is less confusing to developers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2444: HBASE-25086 Refactor Replication: move the default ReplicationSinkSer…
Apache-HBase commented on pull request #2444: URL: https://github.com/apache/hbase/pull/2444#issuecomment-697167038 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 1s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 47s | master passed | | +1 :green_heart: | checkstyle | 1m 34s | master passed | | +1 :green_heart: | spotbugs | 2m 48s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 11s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 45s | the patch passed | | +1 :green_heart: | checkstyle | 0m 23s | The patch passed checkstyle in hbase-common | | +1 :green_heart: | checkstyle | 1m 12s | hbase-server: The patch generated 0 new + 43 unchanged - 3 fixed = 43 total (was 46) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 19s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 3m 7s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 21s | The patch does not generate ASF License warnings. | | | | 38m 36s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2444/5/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2444 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux ff759009d7f8 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / e7797208d6 | | Max. process+thread count | 84 (vs. ulimit of 3) | | modules | C: hbase-common hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2444/5/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 merged pull request #2443: Should not use XXXService.Interface.class.getSimpleName as stub key prefix in AsyncConnectionImpl
Apache9 merged pull request #2443: URL: https://github.com/apache/hbase/pull/2443 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #251: HBASE-22114 Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
Apache-HBase commented on pull request #251: URL: https://github.com/apache/hbase/pull/251#issuecomment-697150208 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 11m 42s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | -1 :x: | hbaseanti | 0m 0s | The patch appears use Hadoop classification instead of HBase. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ branch-1 Compile Tests _ | | +0 :ok: | mvndep | 2m 48s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 8m 14s | branch-1 passed | | +1 :green_heart: | compile | 1m 50s | branch-1 passed with JDK v1.8.0_262 | | +1 :green_heart: | compile | 1m 58s | branch-1 passed with JDK v1.7.0_272 | | +1 :green_heart: | checkstyle | 8m 59s | branch-1 passed | | +0 :ok: | refguide | 4m 14s | branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | shadedjars | 3m 15s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 3m 23s | branch-1 passed with JDK v1.8.0_262 | | +1 :green_heart: | javadoc | 5m 37s | branch-1 passed with JDK v1.7.0_272 | | +0 :ok: | spotbugs | 2m 55s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +0 :ok: | findbugs | 0m 24s | branch/hbase-resource-bundle no findbugs output file (findbugsXml.xml) | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 2m 6s | the patch passed | | -1 :x: | compile | 0m 8s | root in the patch failed with JDK v1.8.0_262. | | -1 :x: | javac | 0m 8s | root in the patch failed with JDK v1.8.0_262. | | +1 :green_heart: | compile | 1m 55s | the patch passed with JDK v1.7.0_272 | | +1 :green_heart: | javac | 1m 55s | the patch passed | | +1 :green_heart: | checkstyle | 5m 51s | root: The patch generated 0 new + 87 unchanged - 11 fixed = 87 total (was 98) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | -1 :x: | xml | 0m 1s | The patch has 5 ill-formed XML file(s). | | +0 :ok: | refguide | 3m 13s | patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect. | | +1 :green_heart: | shadedjars | 3m 5s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | hadoopcheck | 5m 3s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | -1 :x: | javadoc | 0m 8s | root in the patch failed with JDK v1.8.0_262. | | +1 :green_heart: | javadoc | 4m 12s | the patch passed with JDK v1.7.0_272 | | +0 :ok: | findbugs | 0m 13s | hbase-resource-bundle has no data from findbugs | | -1 :x: | findbugs | 0m 8s | hbase-tinylfu-blockcache in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 177m 25s | root in the patch failed. | | +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. | | | | 293m 31s | | | Reason | Tests | |---:|:--| | XML | Parsing Error(s): | | | hbase-common/src/main/resources/hbase-default.xml | | | hbase-it/pom.xml | | | hbase-resource-bundle/src/main/resources/supplemental-models.xml | | | hbase-tinylfu-blockcache/pom.xml | | | pom.xml | | Failed junit tests | hadoop.hbase.mapreduce.TestTableSnapshotInputFormat | | | hadoop.hbase.mapreduce.TestLoadIncrementalHFilesUseSecurityEndPoint | | | hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFiles | | | hadoop.hbase.mapreduce.TestLoadIncrementalHFiles | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-251/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/251 | | Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile refguide xml | | uname | Linux ed75486022a4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-agent/workspace/HBase-PreCommit-GitHub-PR_PR-251/out/precommit/personality/provided.sh | | git rev
[jira] [Commented] (HBASE-25068) Pass WALFactory to Replication so it knows of all WALProviders, not just default/user-space
[ https://issues.apache.org/jira/browse/HBASE-25068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200570#comment-17200570 ] Duo Zhang commented on HBASE-25068: --- Thank you sir. I think on a feature branch we could do changes more aggressively so it will help us move faster. > Pass WALFactory to Replication so it knows of all WALProviders, not just > default/user-space > --- > > Key: HBASE-25068 > URL: https://issues.apache.org/jira/browse/HBASE-25068 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Minor > Fix For: HBASE-18070 > > Attachments: > 0001-HBASE-25068-Pass-WALFactory-to-Replication-so-it-kno.patch.2, > 0001-HBASE-25068-Pass-WALFactory-to-Replication-so-it-kno.patch.master > > > Small change that passes all WALProviders to ReplicationService rather than > just the default/user-space WALProvider. It does this using the WALFactory > vessel since it holds all Providers. This change is to be exploited by > adjacent sub-task HBASE-25055 in follow-on. This sub-task also exists to make > the HBASE-25055 patch smaller and more focused, easier to review. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2442: HBASE-25079 Upgrade Bootstrap to 3.3.7
Apache-HBase commented on pull request #2442: URL: https://github.com/apache/hbase/pull/2442#issuecomment-697061543 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] infraio commented on pull request #2444: HBASE-25086 Refactor Replication: move the default ReplicationSinkSer…
infraio commented on pull request #2444: URL: https://github.com/apache/hbase/pull/2444#issuecomment-697122549 And I removed the ThreadPool and use RS's ChoreService instead. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2444: HBASE-25086 Refactor Replication: move the default ReplicationSinkSer…
Apache-HBase commented on pull request #2444: URL: https://github.com/apache/hbase/pull/2444#issuecomment-697126009 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2443: Should not use XXXService.Interface.class.getSimpleName as stub key prefix in AsyncConnectionImpl
Apache-HBase commented on pull request #2443: URL: https://github.com/apache/hbase/pull/2443#issuecomment-697112542 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell commented on pull request #2442: HBASE-25079 Upgrade Bootstrap to 3.3.7
apurtell commented on pull request #2442: URL: https://github.com/apache/hbase/pull/2442#issuecomment-697081182 Will add RAT exclusions, back soon. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2439: HBASE-25081 Up the container nproc uplimit to 30000
Apache-HBase commented on pull request #2439: URL: https://github.com/apache/hbase/pull/2439#issuecomment-696786684 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack closed pull request #2421: HBASE-25067 Edit of log messages around async WAL Replication; checks…
saintstack closed pull request #2421: URL: https://github.com/apache/hbase/pull/2421 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2353: HBASE-24981 Enable table replication fails from 1.x to 2.x if table a…
Apache-HBase commented on pull request #2353: URL: https://github.com/apache/hbase/pull/2353#issuecomment-696388232 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2438: HBASE-25033 : Provide an option to ignore svn ci for create-release
Apache-HBase commented on pull request #2438: URL: https://github.com/apache/hbase/pull/2438#issuecomment-696703259 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack merged pull request #2435: HBASE-25067 Edit of log messages around async WAL Replication; checks…
saintstack merged pull request #2435: URL: https://github.com/apache/hbase/pull/2435 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] virajjasani commented on pull request #2436: HBASE-25080 Should not use AssignmentManager to test whether a table …
virajjasani commented on pull request #2436: URL: https://github.com/apache/hbase/pull/2436#issuecomment-696557272 +1 for the change, but test failures seem relevant? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack merged pull request #2434: HBASE-25068 Pass WALFactory to Replication so it knows of all WALProv…
saintstack merged pull request #2434: URL: https://github.com/apache/hbase/pull/2434 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2437: HBASE-25053 WAL replay should ignore 0-length files
Apache-HBase commented on pull request #2437: URL: https://github.com/apache/hbase/pull/2437#issuecomment-696496119 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] infraio merged pull request #2430: HBASE-25074 Refactor ReplicationSinkManager: reduce code and make it …
infraio merged pull request #2430: URL: https://github.com/apache/hbase/pull/2430 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #2421: HBASE-25067 Edit of log messages around async WAL Replication; checks…
saintstack commented on pull request #2421: URL: https://github.com/apache/hbase/pull/2421#issuecomment-696858663 Merged to branch-2 manually. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2440: HBASE-25082: Per table WAL metrics: appendCount and appendSize
Apache-HBase commented on pull request #2440: URL: https://github.com/apache/hbase/pull/2440#issuecomment-696933888 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #2434: HBASE-25068 Pass WALFactory to Replication so it knows of all WALProv…
saintstack commented on pull request #2434: URL: https://github.com/apache/hbase/pull/2434#issuecomment-696921949 Ran the failed test locally w/ jdk8 and jdk11 and it passed. It fails 10% of the time according to flakey dashboard. Let me merge. Will keep an eye on it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack merged pull request #2439: HBASE-25081 Up the container nproc uplimit to 30000
saintstack merged pull request #2439: URL: https://github.com/apache/hbase/pull/2439 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2308: HBASE-20598 - Upgrade to JRuby 9.2
Apache-HBase commented on pull request #2308: URL: https://github.com/apache/hbase/pull/2308#issuecomment-696697458 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2441: HBASE-25085 Add support for java properties to hbase-vote.sh
Apache-HBase commented on pull request #2441: URL: https://github.com/apache/hbase/pull/2441#issuecomment-697002166 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] gjacoby126 commented on a change in pull request #2440: HBASE-25082: Per table WAL metrics: appendCount and appendSize
gjacoby126 commented on a change in pull request #2440:
URL: https://github.com/apache/hbase/pull/2440#discussion_r493012475
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsWAL.java
##
@@ -58,9 +59,10 @@ public void postSync(final long timeInNanos, final int
handlerSyncs) {
@Override
public void postAppend(final long size, final long time, final WALKey logkey,
final WALEdit logEdit) throws IOException {
-source.incrementAppendCount();
+TableName tableName = logkey.getTableName();
Review comment:
Is it legal for an implementation to pass in a null WALKey, or a
non-null WALKey with a null TableName? (As was done in the prior version of
TestMetricsWAL?) If so, we may need some NPE protection.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2437: HBASE-25053 WAL replay should ignore 0-length files
Apache9 commented on pull request #2437: URL: https://github.com/apache/hbase/pull/2437#issuecomment-696734732 This is just a simple optimization or something related to correctness? I'm not sure whether at this step we have finished recoverLease for all the WAL files? I mean whether the length in the FileStatus can be trusted? Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2436: HBASE-25080 Should not use AssignmentManager to test whether a table …
Apache-HBase commented on pull request #2436: URL: https://github.com/apache/hbase/pull/2436#issuecomment-696483579 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bharathv commented on a change in pull request #2440: HBASE-25082: Per table WAL metrics: appendCount and appendSize
bharathv commented on a change in pull request #2440:
URL: https://github.com/apache/hbase/pull/2440#discussion_r493099034
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsWAL.java
##
@@ -58,9 +59,10 @@ public void postSync(final long timeInNanos, final int
handlerSyncs) {
@Override
public void postAppend(final long size, final long time, final WALKey logkey,
final WALEdit logEdit) throws IOException {
-source.incrementAppendCount();
+TableName tableName = logkey.getTableName();
Review comment:
Don't think it's optional (the test I fixed seems like an outlier). At
least the protobuf based WAL implementation mandates the table name to be a
required field and there are multiple other places in the code (like
replication tailing etc) that assume a non-null table name.
```
message WALKey {
required bytes encoded_region_name = 1;
required bytes table_name = 2;
```
Wish there is a better way to enforce it but using (non)nullable annotations
is not a common practice in HBase code.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2434: HBASE-25068 Pass WALFactory to Replication so it knows of all WALProv…
Apache-HBase commented on pull request #2434: URL: https://github.com/apache/hbase/pull/2434#issuecomment-696375809 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2435: HBASE-25067 Edit of log messages around async WAL Replication; checks…
Apache-HBase commented on pull request #2435: URL: https://github.com/apache/hbase/pull/2435#issuecomment-696402037 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2326: HBASE-24967 The table.jsp cost long time to load if the table include…
Apache-HBase commented on pull request #2326: URL: https://github.com/apache/hbase/pull/2326#issuecomment-696554325 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bsglz commented on a change in pull request #2326: HBASE-24967 The table.jsp cost long time to load if the table include…
bsglz commented on a change in pull request #2326:
URL: https://github.com/apache/hbase/pull/2326#discussion_r492441835
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
##
@@ -1741,9 +1741,10 @@ RegionLoad createRegionLoad(final HRegion r,
RegionLoad.Builder regionLoadBldr,
.setBlocksLocalWeight(blocksLocalWeight)
.setBlocksLocalWithSsdWeight(blocksLocalWithSsdWeight)
.setBlocksTotalWeight(blocksTotalWeight)
+
.setCompactionState(ProtobufUtil.createCompactionStateForRegionLoad(r.getCompactionState()))
.setLastMajorCompactionTs(r.getOldestHfileTs(true));
r.setCompleteSequenceId(regionLoadBldr);
-
+LOG.info(">>setCompactionState:{}",r.getCompactionState());
Review comment:
Will fix, thanks.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
##
@@ -3907,4 +3909,51 @@ public MetaRegionLocationCache
getMetaRegionLocationCache() {
public RSGroupInfoManager getRSGroupInfoManager() {
return rsGroupInfoManager;
}
+
+ /**
+ * Get the compaction state of the table
+ *
+ * @param tableName The table name
+ * @return CompactionState Compaction state of the table
+ */
+ public org.apache.hadoop.hbase.client.CompactionState getCompactionState(
Review comment:
Conflict with
org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos.GetRegionInfoResponse.CompactionState
##
File path:
hbase-client/src/main/java/org/apache/hadoop/hbase/RegionMetricsBuilder.java
##
@@ -263,6 +268,10 @@ public RegionMetricsBuilder setBlocksTotalWeight(long
value) {
this.blocksTotalWeight = value;
return this;
}
+ public RegionMetricsBuilder setCompactionState(CompactionState
compactionState) {
+this.compactionState = compactionState;
+return this;
+ }
Review comment:
Ok, will fix.
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactionState.java
##
@@ -164,10 +166,10 @@ private void compaction(final String tableName, final int
flushes,
long curt = System.currentTimeMillis();
long waitTime = 5000;
long endt = curt + waitTime;
- CompactionState state = admin.getCompactionState(table);
+ CompactionState state = master.getCompactionState(table);
Review comment:
Make sense, will fix later, thanks.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
##
@@ -3907,4 +3909,51 @@ public MetaRegionLocationCache
getMetaRegionLocationCache() {
public RSGroupInfoManager getRSGroupInfoManager() {
return rsGroupInfoManager;
}
+
+ /**
+ * Get the compaction state of the table
+ *
+ * @param tableName The table name
+ * @return CompactionState Compaction state of the table
+ */
+ public org.apache.hadoop.hbase.client.CompactionState getCompactionState(
Review comment:
Good idea, will fix, thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] nkalmar commented on pull request #2308: HBASE-20598 - Upgrade to JRuby 9.2
nkalmar commented on pull request #2308: URL: https://github.com/apache/hbase/pull/2308#issuecomment-696685975 Rebased to current master This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] wchevreuil commented on a change in pull request #2430: HBASE-25074 Refactor ReplicationSinkManager: reduce code and make it …
wchevreuil commented on a change in pull request #2430:
URL: https://github.com/apache/hbase/pull/2430#discussion_r492588627
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HBaseReplicationEndpoint.java
##
@@ -63,7 +121,7 @@ protected synchronized void disconnect() {
* A private method used to re-establish a zookeeper session with a peer
cluster.
* @param ke
*/
- protected void reconnect(KeeperException ke) {
+ private void reconnect(KeeperException ke) {
Review comment:
> So if one user want to use these logics, it is better to copy these
code to their implementation.
It's not that simple, as it would actually require reimplement all other
methods currently referring these private ones, leading to a reimplementation
of the whole class.
I agree with @Apache9 , if we are to guarantee a certain level of
accessibility here, we should rather discuss about the interfaces audiences. I
guess this would be rather a topic for a separate jira/PR, don't want to
deviate this current one from it's original purpose.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] joshelser commented on pull request #2308: HBASE-20598 - Upgrade to JRuby 9.2
joshelser commented on pull request #2308: URL: https://github.com/apache/hbase/pull/2308#issuecomment-696870760 Some subtle changes I'm noticing, the prompt is different. Now.. ``` hbase(#):003:0> ``` and before... ``` hbase(main):001:0> ``` Timestamps are also pretty-printed now: ``` ROW COLUMN+CELL r1 column=f:q, timestamp=2020-09-22T13:33:56.089, value=value 1 row(s) Took 0.1200 seconds ``` was before.. ``` hbase(main):001:0> scan 'j1' ROW COLUMN+CELL r1 column=f:q, timestamp=1600796036089, value=value 1 row(s) Took 0.3820 seconds ``` The timestamp one is probably something we should fix. I think the prompt fixing would be nice to fix, but I'm not sure how to do that without a deep dive :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2419: HBASE-25050 - We initialize Filesystems more than once.
Apache-HBase commented on pull request #2419: URL: https://github.com/apache/hbase/pull/2419#issuecomment-696690164 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2430: HBASE-25074 Refactor ReplicationSinkManager: reduce code and make it …
Apache-HBase commented on pull request #2430: URL: https://github.com/apache/hbase/pull/2430#issuecomment-696477637 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2430: HBASE-25074 Refactor ReplicationSinkManager: reduce code and make it …
infraio commented on a change in pull request #2430:
URL: https://github.com/apache/hbase/pull/2430#discussion_r493111540
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HBaseReplicationEndpoint.java
##
@@ -63,7 +121,7 @@ protected synchronized void disconnect() {
* A private method used to re-establish a zookeeper session with a peer
cluster.
* @param ke
*/
- protected void reconnect(KeeperException ke) {
+ private void reconnect(KeeperException ke) {
Review comment:
There is a issue about this, see HBASE-15982.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2385: HBASE-24950 Another solution for Splittable Meta
Apache-HBase commented on pull request #2385: URL: https://github.com/apache/hbase/pull/2385#issuecomment-696786699 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on a change in pull request #2326: HBASE-24967 The table.jsp cost long time to load if the table include…
Apache9 commented on a change in pull request #2326:
URL: https://github.com/apache/hbase/pull/2326#discussion_r492448370
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
##
@@ -3907,4 +3909,51 @@ public MetaRegionLocationCache
getMetaRegionLocationCache() {
public RSGroupInfoManager getRSGroupInfoManager() {
return rsGroupInfoManager;
}
+
+ /**
+ * Get the compaction state of the table
+ *
+ * @param tableName The table name
+ * @return CompactionState Compaction state of the table
+ */
+ public org.apache.hadoop.hbase.client.CompactionState getCompactionState(
Review comment:
Then better not import
org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos.GetRegionInfoResponse.CompactionState
directly? Import
org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos.GetRegionInfoResponse,
and then use GetRegionInfoResponse.CompactionState to reference the protobuf
message.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-18070) Enable memstore replication for meta replica
[ https://issues.apache.org/jira/browse/HBASE-18070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200499#comment-17200499 ] Michael Stack commented on HBASE-18070: --- Made a HBASE-18070 branch just now from master at e7797208d6ca10a10d37b77591e1f0531ed57dfc Applied HBASE-25068 to HBASE-18070. > Enable memstore replication for meta replica > > > Key: HBASE-18070 > URL: https://issues.apache.org/jira/browse/HBASE-18070 > Project: HBase > Issue Type: New Feature >Reporter: Hua Xiang >Assignee: Huaxiang Sun >Priority: Major > > Based on the current doc, memstore replication is not enabled for meta > replica. Memstore replication will be a good improvement for meta replica. > Create jira to track this effort (feasibility, design, implementation, etc). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25068) Pass WALFactory to Replication so it knows of all WALProviders, not just default/user-space
[ https://issues.apache.org/jira/browse/HBASE-25068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200498#comment-17200498 ] Michael Stack commented on HBASE-25068: --- Pushed master branch patch on to HBASE-18070 which I just branched from master. > Pass WALFactory to Replication so it knows of all WALProviders, not just > default/user-space > --- > > Key: HBASE-25068 > URL: https://issues.apache.org/jira/browse/HBASE-25068 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Minor > Fix For: HBASE-18070 > > Attachments: > 0001-HBASE-25068-Pass-WALFactory-to-Replication-so-it-kno.patch.2, > 0001-HBASE-25068-Pass-WALFactory-to-Replication-so-it-kno.patch.master > > > Small change that passes all WALProviders to ReplicationService rather than > just the default/user-space WALProvider. It does this using the WALFactory > vessel since it holds all Providers. This change is to be exploited by > adjacent sub-task HBASE-25055 in follow-on. This sub-task also exists to make > the HBASE-25055 patch smaller and more focused, easier to review. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25068) Pass WALFactory to Replication so it knows of all WALProviders, not just default/user-space
[ https://issues.apache.org/jira/browse/HBASE-25068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-25068: -- Fix Version/s: (was: 2.4.0) (was: 3.0.0-alpha-1) HBASE-18070 > Pass WALFactory to Replication so it knows of all WALProviders, not just > default/user-space > --- > > Key: HBASE-25068 > URL: https://issues.apache.org/jira/browse/HBASE-25068 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Minor > Fix For: HBASE-18070 > > Attachments: > 0001-HBASE-25068-Pass-WALFactory-to-Replication-so-it-kno.patch.2, > 0001-HBASE-25068-Pass-WALFactory-to-Replication-so-it-kno.patch.master > > > Small change that passes all WALProviders to ReplicationService rather than > just the default/user-space WALProvider. It does this using the WALFactory > vessel since it holds all Providers. This change is to be exploited by > adjacent sub-task HBASE-25055 in follow-on. This sub-task also exists to make > the HBASE-25055 patch smaller and more focused, easier to review. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25068) Pass WALFactory to Replication so it knows of all WALProviders, not just default/user-space
[ https://issues.apache.org/jira/browse/HBASE-25068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-25068: -- Attachment: 0001-HBASE-25068-Pass-WALFactory-to-Replication-so-it-kno.patch.master > Pass WALFactory to Replication so it knows of all WALProviders, not just > default/user-space > --- > > Key: HBASE-25068 > URL: https://issues.apache.org/jira/browse/HBASE-25068 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.4.0 > > Attachments: > 0001-HBASE-25068-Pass-WALFactory-to-Replication-so-it-kno.patch.2, > 0001-HBASE-25068-Pass-WALFactory-to-Replication-so-it-kno.patch.master > > > Small change that passes all WALProviders to ReplicationService rather than > just the default/user-space WALProvider. It does this using the WALFactory > vessel since it holds all Providers. This change is to be exploited by > adjacent sub-task HBASE-25055 in follow-on. This sub-task also exists to make > the HBASE-25055 patch smaller and more focused, easier to review. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25068) Pass WALFactory to Replication so it knows of all WALProviders, not just default/user-space
[ https://issues.apache.org/jira/browse/HBASE-25068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack updated HBASE-25068: -- Attachment: 0001-HBASE-25068-Pass-WALFactory-to-Replication-so-it-kno.patch.2 > Pass WALFactory to Replication so it knows of all WALProviders, not just > default/user-space > --- > > Key: HBASE-25068 > URL: https://issues.apache.org/jira/browse/HBASE-25068 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.4.0 > > Attachments: > 0001-HBASE-25068-Pass-WALFactory-to-Replication-so-it-kno.patch.2 > > > Small change that passes all WALProviders to ReplicationService rather than > just the default/user-space WALProvider. It does this using the WALFactory > vessel since it holds all Providers. This change is to be exploited by > adjacent sub-task HBASE-25055 in follow-on. This sub-task also exists to make > the HBASE-25055 patch smaller and more focused, easier to review. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] infraio opened a new pull request #2444: HBASE-25086 Refactor Replication: move the default ReplicationSinkSer…
infraio opened a new pull request #2444: URL: https://github.com/apache/hbase/pull/2444 …vice implementation out This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25068) Pass WALFactory to Replication so it knows of all WALProviders, not just default/user-space
[ https://issues.apache.org/jira/browse/HBASE-25068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200494#comment-17200494 ] Michael Stack commented on HBASE-25068: --- OK. No problem. Will revert and make a HBASE-18070 branch. > Pass WALFactory to Replication so it knows of all WALProviders, not just > default/user-space > --- > > Key: HBASE-25068 > URL: https://issues.apache.org/jira/browse/HBASE-25068 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Small change that passes all WALProviders to ReplicationService rather than > just the default/user-space WALProvider. It does this using the WALFactory > vessel since it holds all Providers. This change is to be exploited by > adjacent sub-task HBASE-25055 in follow-on. This sub-task also exists to make > the HBASE-25055 patch smaller and more focused, easier to review. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25086) Refactor Replication: move the default ReplicationSinkService implementation out
[ https://issues.apache.org/jira/browse/HBASE-25086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-25086: --- Description: Now the Replication implements both ReplicationSourceService and ReplicationSinkService interface. But most of code is only related to ReplicationSourceService. Meanwhile, there is a bug when HRegionServer#buildServerLoad, as it not consider the case: ReplicationSourceService implementation is different with ReplicationSinkService. (was: Now the Replication implements both ReplicationSourceService and ReplicationSinkService interface. But most of code is only related to ReplicationSourceService. Meanwhile, there is a bug when buildServerLoad, as it not consider the case: ReplicationSourceService implementation is different with ReplicationSinkService.) > Refactor Replication: move the default ReplicationSinkService implementation > out > > > Key: HBASE-25086 > URL: https://issues.apache.org/jira/browse/HBASE-25086 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Priority: Major > > Now the Replication implements both ReplicationSourceService and > ReplicationSinkService interface. But most of code is only related to > ReplicationSourceService. Meanwhile, there is a bug when > HRegionServer#buildServerLoad, as it not consider the case: > ReplicationSourceService implementation is different with > ReplicationSinkService. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25086) Refactor Replication: move the default ReplicationSinkService implementation out
Guanghao Zhang created HBASE-25086: -- Summary: Refactor Replication: move the default ReplicationSinkService implementation out Key: HBASE-25086 URL: https://issues.apache.org/jira/browse/HBASE-25086 Project: HBase Issue Type: Improvement Reporter: Guanghao Zhang Now the Replication implements both ReplicationSourceService and ReplicationSinkService interface. But most of code is only related to ReplicationSourceService. Meanwhile, there is a bug when buildServerLoad, as it not consider the case: ReplicationSourceService implementation is different with ReplicationSinkService. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2442: HBASE-25079 Upgrade Bootstrap to 3.3.7
Apache-HBase commented on pull request #2442: URL: https://github.com/apache/hbase/pull/2442#issuecomment-697114351 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 8s | master passed | | -0 :warning: | javadoc | 0m 40s | hbase-server in master failed. | | -0 :warning: | javadoc | 0m 51s | hbase-thrift in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 12s | the patch passed | | -0 :warning: | javadoc | 0m 42s | hbase-server in the patch failed. | | -0 :warning: | javadoc | 0m 53s | hbase-thrift in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 134m 51s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 3m 54s | hbase-thrift in the patch passed. | | | | 153m 26s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2442/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2442 | | Optional Tests | javac javadoc unit | | uname | Linux af228a11495d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7e910a573f | | Default Java | 2020-01-14 | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2442/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2442/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-thrift.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2442/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2442/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-thrift.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2442/1/testReport/ | | Max. process+thread count | 4366 (vs. ulimit of 3) | | modules | C: hbase-server hbase-thrift U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2442/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2443: Should not use XXXService.Interface.class.getSimpleName as stub key prefix in AsyncConnectionImpl
Apache-HBase commented on pull request #2443: URL: https://github.com/apache/hbase/pull/2443#issuecomment-697113046 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 40s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 49s | master passed | | +1 :green_heart: | checkstyle | 0m 28s | master passed | | +1 :green_heart: | spotbugs | 1m 1s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 24s | the patch passed | | +1 :green_heart: | checkstyle | 0m 26s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 25s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 1m 5s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. | | | | 30m 52s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2443/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2443 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 5ac2738217e1 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7e910a573f | | Max. process+thread count | 94 (vs. ulimit of 3) | | modules | C: hbase-client U: hbase-client | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2443/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2443: Should not use XXXService.Interface.class.getSimpleName as stub key prefix in AsyncConnectionImpl
Apache-HBase commented on pull request #2443: URL: https://github.com/apache/hbase/pull/2443#issuecomment-697112703 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 1s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 47s | master passed | | +1 :green_heart: | compile | 0m 30s | master passed | | +1 :green_heart: | shadedjars | 7m 23s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 27s | hbase-client in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 35s | the patch passed | | +1 :green_heart: | compile | 0m 29s | the patch passed | | +1 :green_heart: | javac | 0m 29s | the patch passed | | +1 :green_heart: | shadedjars | 7m 22s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 25s | hbase-client in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 22s | hbase-client in the patch passed. | | | | 29m 31s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2443/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2443 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 8e90dad9d5ce 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7e910a573f | | Default Java | 2020-01-14 | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2443/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2443/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2443/1/testReport/ | | Max. process+thread count | 201 (vs. ulimit of 3) | | modules | C: hbase-client U: hbase-client | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2443/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2443: Should not use XXXService.Interface.class.getSimpleName as stub key prefix in AsyncConnectionImpl
Apache-HBase commented on pull request #2443: URL: https://github.com/apache/hbase/pull/2443#issuecomment-697112542 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 42s | master passed | | +1 :green_heart: | compile | 0m 26s | master passed | | +1 :green_heart: | shadedjars | 6m 58s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 26s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 21s | the patch passed | | +1 :green_heart: | compile | 0m 32s | the patch passed | | +1 :green_heart: | javac | 0m 32s | the patch passed | | +1 :green_heart: | shadedjars | 8m 5s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 14s | hbase-client in the patch passed. | | | | 28m 35s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2443/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2443 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 24425c33c42d 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7e910a573f | | Default Java | 1.8.0_232 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2443/1/testReport/ | | Max. process+thread count | 346 (vs. ulimit of 3) | | modules | C: hbase-client U: hbase-client | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2443/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] wwmandy opened a new pull request #2443: Should not use XXXService.Interface.class.getSimpleName as stub key prefix in AsyncConnectionImpl
wwmandy opened a new pull request #2443: URL: https://github.com/apache/hbase/pull/2443 It will just return "Interface" for all services. This is not a bug for AsyncConnectionImpl as we have different maps for ClientService and AdminService, but it is still a bit confusing to developers. I suggest either we just use empty string and add a comment to say that the service name is not important here, or change to use XXXService.class.getSimpleName() or XXXService.getDescriptor().getName(), so it is less confusing to developers. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell commented on pull request #2442: HBASE-25079 Upgrade Bootstrap to 3.3.7
apurtell commented on pull request #2442: URL: https://github.com/apache/hbase/pull/2442#issuecomment-697081182 Will add RAT exclusions, back soon. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-25079) Upgrade Bootstrap to 3.3.7
[ https://issues.apache.org/jira/browse/HBASE-25079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25079: Status: Patch Available (was: Open) > Upgrade Bootstrap to 3.3.7 > -- > > Key: HBASE-25079 > URL: https://issues.apache.org/jira/browse/HBASE-25079 > Project: HBase > Issue Type: Improvement > Components: security, UI > Environment: ad >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.3, 1.7.0, 2.4.0, 2.2.7 > > > Our UI embeds Bootstrap 3.0.0. There are some reported security issues. > Upgrade to Bootstrap 3.3.7. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25068) Pass WALFactory to Replication so it knows of all WALProviders, not just default/user-space
[
https://issues.apache.org/jira/browse/HBASE-25068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200465#comment-17200465
]
Duo Zhang commented on HBASE-25068:
---
Hi, sir, you missed my comments on the PR?
{quote}
Let's change the base branch to master so we can commit the changes to a
feature branch first? The patch is small and OK, but at least without the
follow-on patches for meta replication, itself is not useful for our current
code base...
{quote}
I said it is better to commit the patch to a feature branch first as it is not
useful for our current code base...
> Pass WALFactory to Replication so it knows of all WALProviders, not just
> default/user-space
> ---
>
> Key: HBASE-25068
> URL: https://issues.apache.org/jira/browse/HBASE-25068
> Project: HBase
> Issue Type: Sub-task
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.4.0
>
>
> Small change that passes all WALProviders to ReplicationService rather than
> just the default/user-space WALProvider. It does this using the WALFactory
> vessel since it holds all Providers. This change is to be exploited by
> adjacent sub-task HBASE-25055 in follow-on. This sub-task also exists to make
> the HBASE-25055 patch smaller and more focused, easier to review.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2442: HBASE-25079 Upgrade Bootstrap to 3.3.7
Apache-HBase commented on pull request #2442: URL: https://github.com/apache/hbase/pull/2442#issuecomment-697061543 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | jshint | 0m 0s | jshint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 8s | Maven dependency ordering for patch | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | -1 :x: | asflicense | 0m 21s | The patch generated 6 ASF License warnings. | | | | 2m 41s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2442/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2442 | | Optional Tests | dupname asflicense jshint | | uname | Linux 61c2447da7f5 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 7e910a573f | | asflicense | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2442/1/artifact/yetus-general-check/output/patch-asflicense-problems.txt | | Max. process+thread count | 48 (vs. ulimit of 3) | | modules | C: hbase-server hbase-thrift U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2442/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell opened a new pull request #2442: HBASE-25079 Upgrade Bootstrap to 3.3.7
apurtell opened a new pull request #2442: URL: https://github.com/apache/hbase/pull/2442 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] infraio commented on a change in pull request #2430: HBASE-25074 Refactor ReplicationSinkManager: reduce code and make it …
infraio commented on a change in pull request #2430:
URL: https://github.com/apache/hbase/pull/2430#discussion_r493111540
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HBaseReplicationEndpoint.java
##
@@ -63,7 +121,7 @@ protected synchronized void disconnect() {
* A private method used to re-establish a zookeeper session with a peer
cluster.
* @param ke
*/
- protected void reconnect(KeeperException ke) {
+ private void reconnect(KeeperException ke) {
Review comment:
There is a issue about this, see HBASE-15982.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2440: HBASE-25082: Per table WAL metrics: appendCount and appendSize
Apache-HBase commented on pull request #2440: URL: https://github.com/apache/hbase/pull/2440#issuecomment-697056723 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 4s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 39s | master passed | | +1 :green_heart: | compile | 1m 14s | master passed | | +1 :green_heart: | shadedjars | 6m 35s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 56s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 26s | the patch passed | | +1 :green_heart: | compile | 1m 14s | the patch passed | | +1 :green_heart: | javac | 1m 14s | the patch passed | | +1 :green_heart: | shadedjars | 6m 36s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 54s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 34s | hbase-hadoop-compat in the patch passed. | | -1 :x: | unit | 145m 34s | hbase-server in the patch failed. | | | | 174m 47s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2440/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2440 | | JIRA Issue | HBASE-25082 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 46ab74cee8f4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c5055f81a | | Default Java | 1.8.0_232 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2440/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2440/2/testReport/ | | Max. process+thread count | 4221 (vs. ulimit of 3) | | modules | C: hbase-hadoop-compat hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2440/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2440: HBASE-25082: Per table WAL metrics: appendCount and appendSize
Apache-HBase commented on pull request #2440: URL: https://github.com/apache/hbase/pull/2440#issuecomment-697053207 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 11s | master passed | | +1 :green_heart: | compile | 1m 26s | master passed | | +1 :green_heart: | shadedjars | 6m 39s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 22s | hbase-hadoop-compat in master failed. | | -0 :warning: | javadoc | 0m 40s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 6s | the patch passed | | +1 :green_heart: | compile | 1m 26s | the patch passed | | +1 :green_heart: | javac | 1m 26s | the patch passed | | +1 :green_heart: | shadedjars | 6m 43s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 19s | hbase-hadoop-compat in the patch failed. | | -0 :warning: | javadoc | 0m 40s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 35s | hbase-hadoop-compat in the patch passed. | | +1 :green_heart: | unit | 129m 7s | hbase-server in the patch passed. | | | | 160m 53s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2440/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2440 | | JIRA Issue | HBASE-25082 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 330709082d70 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c5055f81a | | Default Java | 2020-01-14 | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2440/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-hadoop-compat.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2440/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2440/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-hadoop-compat.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2440/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2440/2/testReport/ | | Max. process+thread count | 4656 (vs. ulimit of 3) | | modules | C: hbase-hadoop-compat hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2440/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] infraio merged pull request #2430: HBASE-25074 Refactor ReplicationSinkManager: reduce code and make it …
infraio merged pull request #2430: URL: https://github.com/apache/hbase/pull/2430 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bharathv commented on a change in pull request #2440: HBASE-25082: Per table WAL metrics: appendCount and appendSize
bharathv commented on a change in pull request #2440:
URL: https://github.com/apache/hbase/pull/2440#discussion_r493099034
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsWAL.java
##
@@ -58,9 +59,10 @@ public void postSync(final long timeInNanos, final int
handlerSyncs) {
@Override
public void postAppend(final long size, final long time, final WALKey logkey,
final WALEdit logEdit) throws IOException {
-source.incrementAppendCount();
+TableName tableName = logkey.getTableName();
Review comment:
Don't think it's optional (the test I fixed seems like an outlier). At
least the protobuf based WAL implementation mandates the table name to be a
required field and there are multiple other places in the code (like
replication tailing etc) that assume a non-null table name.
```
message WALKey {
required bytes encoded_region_name = 1;
required bytes table_name = 2;
```
Wish there is a better way to enforce it but using (non)nullable annotations
is not a common practice in HBase code.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2440: HBASE-25082: Per table WAL metrics: appendCount and appendSize
Apache-HBase commented on pull request #2440: URL: https://github.com/apache/hbase/pull/2440#issuecomment-697015514 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 22s | master passed | | +1 :green_heart: | checkstyle | 1m 17s | master passed | | +1 :green_heart: | spotbugs | 2m 28s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 24s | the patch passed | | +1 :green_heart: | checkstyle | 1m 17s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 6s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 47s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 25s | The patch does not generate ASF License warnings. | | | | 34m 35s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2440/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2440 | | JIRA Issue | HBASE-25082 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 315b892f6ece 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c5055f81a | | Max. process+thread count | 94 (vs. ulimit of 3) | | modules | C: hbase-hadoop-compat hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2440/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2441: HBASE-25085 Add support for java properties to hbase-vote.sh
Apache-HBase commented on pull request #2441: URL: https://github.com/apache/hbase/pull/2441#issuecomment-697003597 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 4m 5s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | shelldocs | 0m 0s | Shelldocs was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 8s | Maven dependency ordering for patch | | +1 :green_heart: | shellcheck | 0m 0s | There were no new shellcheck issues. | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | +0 :ok: | asflicense | 0m 0s | ASF License check generated no output? | | | | 5m 30s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2441/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2441 | | Optional Tests | dupname asflicense shellcheck shelldocs | | uname | Linux 97b9e39df5a0 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 2c5055f81a | | Max. process+thread count | 52 (vs. ulimit of 3) | | modules | C: U: | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2441/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) shellcheck=0.4.6 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2441: HBASE-25085 Add support for java properties to hbase-vote.sh
Apache-HBase commented on pull request #2441: URL: https://github.com/apache/hbase/pull/2441#issuecomment-697002166 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2440: HBASE-25082: Per table WAL metrics: appendCount and appendSize
Apache-HBase commented on pull request #2440: URL: https://github.com/apache/hbase/pull/2440#issuecomment-697001679 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 3s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 39s | master passed | | +1 :green_heart: | compile | 1m 13s | master passed | | +1 :green_heart: | shadedjars | 6m 33s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 56s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 28s | the patch passed | | +1 :green_heart: | compile | 1m 17s | the patch passed | | +1 :green_heart: | javac | 1m 17s | the patch passed | | +1 :green_heart: | shadedjars | 6m 33s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 55s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 34s | hbase-hadoop-compat in the patch passed. | | +1 :green_heart: | unit | 146m 10s | hbase-server in the patch passed. | | | | 175m 31s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2440/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2440 | | JIRA Issue | HBASE-25082 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 1f84d2f75bf3 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 17ebf917ba | | Default Java | 1.8.0_232 | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2440/1/testReport/ | | Max. process+thread count | 4365 (vs. ulimit of 3) | | modules | C: hbase-hadoop-compat hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2440/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-25084) RegexStringComparator in ParseFilter should support case-insensitive regexes
[ https://issues.apache.org/jira/browse/HBASE-25084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200401#comment-17200401 ] Scott Hunt commented on HBASE-25084: I've added a very simple patch, as a potential starting point. Partially looking for thoughts on the desired approach to add stuff to ParseFilter. > RegexStringComparator in ParseFilter should support case-insensitive regexes > > > Key: HBASE-25084 > URL: https://issues.apache.org/jira/browse/HBASE-25084 > Project: HBase > Issue Type: Improvement > Components: Thrift >Reporter: Scott Hunt >Priority: Major > Attachments: HBASE-25084.v0.patch > > > I would like to be able to set CASE_INSENSITIVE to RegexStringComparator > (using thrift.) There doesn't seem (currently) to be a way to do that. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25084) RegexStringComparator in ParseFilter should support case-insensitive regexes
[ https://issues.apache.org/jira/browse/HBASE-25084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Hunt updated HBASE-25084: --- Attachment: HBASE-25084.v0.patch > RegexStringComparator in ParseFilter should support case-insensitive regexes > > > Key: HBASE-25084 > URL: https://issues.apache.org/jira/browse/HBASE-25084 > Project: HBase > Issue Type: Improvement > Components: Thrift >Reporter: Scott Hunt >Priority: Major > Attachments: HBASE-25084.v0.patch > > > I would like to be able to set CASE_INSENSITIVE to RegexStringComparator > (using thrift.) There doesn't seem (currently) to be a way to do that. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] ndimiduk opened a new pull request #2441: HBASE-25085 Add support for java properties to hbase-vote.sh
ndimiduk opened a new pull request #2441: URL: https://github.com/apache/hbase/pull/2441 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-25085) Add support for java properties to hbase-vote.sh
Nick Dimiduk created HBASE-25085:
Summary: Add support for java properties to hbase-vote.sh
Key: HBASE-25085
URL: https://issues.apache.org/jira/browse/HBASE-25085
Project: HBase
Issue Type: Task
Components: community
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Fix For: 3.0.0-alpha-1
A small tweak to {{hbase-vote.sh}} so that it can accept a -D argument. I want
this for evaluating branch-2.3 with Hadoop3 and JDK11.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2440: HBASE-25082: Per table WAL metrics: appendCount and appendSize
Apache-HBase commented on pull request #2440: URL: https://github.com/apache/hbase/pull/2440#issuecomment-696998814 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 20s | master passed | | +1 :green_heart: | compile | 1m 33s | master passed | | +1 :green_heart: | shadedjars | 7m 10s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 21s | hbase-hadoop-compat in master failed. | | -0 :warning: | javadoc | 0m 42s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 6s | the patch passed | | +1 :green_heart: | compile | 1m 36s | the patch passed | | +1 :green_heart: | javac | 1m 36s | the patch passed | | +1 :green_heart: | shadedjars | 7m 21s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 20s | hbase-hadoop-compat in the patch failed. | | -0 :warning: | javadoc | 0m 44s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 43s | hbase-hadoop-compat in the patch passed. | | +1 :green_heart: | unit | 136m 20s | hbase-server in the patch passed. | | | | 168m 30s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2440/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2440 | | JIRA Issue | HBASE-25082 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux c397bfb664b7 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 17ebf917ba | | Default Java | 2020-01-14 | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2440/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-hadoop-compat.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2440/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2440/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-hadoop-compat.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2440/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2440/1/testReport/ | | Max. process+thread count | 4214 (vs. ulimit of 3) | | modules | C: hbase-hadoop-compat hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2440/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-25084) RegexStringComparator in ParseFilter should support case-insensitive regexes
Scott Hunt created HBASE-25084: -- Summary: RegexStringComparator in ParseFilter should support case-insensitive regexes Key: HBASE-25084 URL: https://issues.apache.org/jira/browse/HBASE-25084 Project: HBase Issue Type: Improvement Components: Thrift Reporter: Scott Hunt I would like to be able to set CASE_INSENSITIVE to RegexStringComparator (using thrift.) There doesn't seem (currently) to be a way to do that. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] gjacoby126 commented on a change in pull request #2440: HBASE-25082: Per table WAL metrics: appendCount and appendSize
gjacoby126 commented on a change in pull request #2440:
URL: https://github.com/apache/hbase/pull/2440#discussion_r493012475
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/MetricsWAL.java
##
@@ -58,9 +59,10 @@ public void postSync(final long timeInNanos, final int
handlerSyncs) {
@Override
public void postAppend(final long size, final long time, final WALKey logkey,
final WALEdit logEdit) throws IOException {
-source.incrementAppendCount();
+TableName tableName = logkey.getTableName();
Review comment:
Is it legal for an implementation to pass in a null WALKey, or a
non-null WALKey with a null TableName? (As was done in the prior version of
TestMetricsWAL?) If so, we may need some NPE protection.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200361#comment-17200361
]
Vladimir Rodionov commented on HBASE-23887:
---
{quote}
Thats why this feature will work when the data really can't fit into BlockCache
-> eviction rate really work hard and it usually means reading blocks evenly
distributed.
{quote}
For such use cases (if they exist in a wild) cache is no help and must be
disabled. HBase can do it per table/cf. I do not see any improvements in this
"feature". It is just ""use cache slightly when data is not catchable" type of
improvemt.
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png,
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, scan_and_gets2.png, wave.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
> hbase.lru.cache.heavy.eviction.bytes.size.limit - set how many bytes have to
> evicted each time that start to avoid of putting data to BlockCache
> By default: if 10 times (100 secunds) evicted more than 10 MB (each time)
> then we start to skip 50% of data blocks.
> When heavy evitions process end then new logic off and will put into
> BlockCache all blocks again.
>
> Descriptions of the test:
> 4 nodes E5-2698 v4 @ 2.20GHz, 700 Gb Mem.
> 4 RegionServers
> 4 tables by 64 regions by 1.88 Gb data in each = 600 Gb total (only FAST_DIFF)
> Total BlockCache Size = 48 Gb (8 % of data in HFiles)
> Random read in 20 threads
>
> I am going to ma
[jira] [Created] (HBASE-25083) make sure the next hbase 1.y release has Hadoop 2.10 as a minimum version
Sean Busbey created HBASE-25083: --- Summary: make sure the next hbase 1.y release has Hadoop 2.10 as a minimum version Key: HBASE-25083 URL: https://issues.apache.org/jira/browse/HBASE-25083 Project: HBase Issue Type: Task Components: documentation, hadoop2 Reporter: Sean Busbey Fix For: 1.7.0 Our reference guide list of prerequisites still has Hadoop 2.8 and 2.9 listed for HBase 1 releases. * [hadoop 2.8 is EOM|https://lists.apache.org/thread.html/r348f7bc93a522f05b7cce78a911854d128a6b1b8bd8124bad4d06ce6%40%3Cuser.hadoop.apache.org%3E] * [hadoop 2.9 is EOM|https://lists.apache.org/thread.html/r16b14cce9504f7a9d228612c6b808e72d8dd20863c78be51a7e04ed5%40%3Cuser.hadoop.apache.org%3E] The current list in the reference guide for HBase 1.6 is just the 1.5 list copied. we should update it to remove 2.8 and 2.9 and make sure we're no longer doing build/test based on those versions for branch-1. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2440: HBASE-25082: Per table WAL metrics: appendCount and appendSize
Apache-HBase commented on pull request #2440: URL: https://github.com/apache/hbase/pull/2440#issuecomment-696933888 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 26s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 41s | master passed | | +1 :green_heart: | checkstyle | 1m 18s | master passed | | +1 :green_heart: | spotbugs | 2m 30s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 26s | the patch passed | | +1 :green_heart: | checkstyle | 1m 15s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 11m 18s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 52s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 26s | The patch does not generate ASF License warnings. | | | | 35m 26s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2440/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2440 | | JIRA Issue | HBASE-25082 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 1233a0a46eeb 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 17ebf917ba | | Max. process+thread count | 94 (vs. ulimit of 3) | | modules | C: hbase-hadoop-compat hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2440/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-25068) Pass WALFactory to Replication so it knows of all WALProviders, not just default/user-space
[ https://issues.apache.org/jira/browse/HBASE-25068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-25068. --- Fix Version/s: 3.0.0-alpha-1 Hadoop Flags: Reviewed Resolution: Fixed Pushed to branch-2 and master. Thanks for review [~zhangduo] > Pass WALFactory to Replication so it knows of all WALProviders, not just > default/user-space > --- > > Key: HBASE-25068 > URL: https://issues.apache.org/jira/browse/HBASE-25068 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Small change that passes all WALProviders to ReplicationService rather than > just the default/user-space WALProvider. It does this using the WALFactory > vessel since it holds all Providers. This change is to be exploited by > adjacent sub-task HBASE-25055 in follow-on. This sub-task also exists to make > the HBASE-25055 patch smaller and more focused, easier to review. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack merged pull request #2434: HBASE-25068 Pass WALFactory to Replication so it knows of all WALProv…
saintstack merged pull request #2434: URL: https://github.com/apache/hbase/pull/2434 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #2434: HBASE-25068 Pass WALFactory to Replication so it knows of all WALProv…
saintstack commented on pull request #2434: URL: https://github.com/apache/hbase/pull/2434#issuecomment-696921949 Ran the failed test locally w/ jdk8 and jdk11 and it passed. It fails 10% of the time according to flakey dashboard. Let me merge. Will keep an eye on it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bharathv opened a new pull request #2440: HBASE-25082: Per table WAL metrics: appendCount and appendSize
bharathv opened a new pull request #2440: URL: https://github.com/apache/hbase/pull/2440 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-25082) Per table WAL metrics: appendCount and appendSize
Bharath Vissapragada created HBASE-25082: Summary: Per table WAL metrics: appendCount and appendSize Key: HBASE-25082 URL: https://issues.apache.org/jira/browse/HBASE-25082 Project: HBase Issue Type: Improvement Components: metrics Reporter: Bharath Vissapragada Assignee: Bharath Vissapragada We were trying to drill down into WAL events at a table scope and figured that appendCount and appendSize are not exposed at a table scope. This would be a helpful metric. I ran the WAL benchmark tool with 10 threads and didn't see any measurable performance difference with the attached patch. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25082) Per table WAL metrics: appendCount and appendSize
[ https://issues.apache.org/jira/browse/HBASE-25082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharath Vissapragada updated HBASE-25082: - Affects Version/s: 2.4.0 1.7.0 2.3.3 3.0.0-alpha-1 > Per table WAL metrics: appendCount and appendSize > - > > Key: HBASE-25082 > URL: https://issues.apache.org/jira/browse/HBASE-25082 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 3.0.0-alpha-1, 2.3.3, 1.7.0, 2.4.0 >Reporter: Bharath Vissapragada >Assignee: Bharath Vissapragada >Priority: Major > Labels: observability > > We were trying to drill down into WAL events at a table scope and figured > that appendCount and appendSize are not exposed at a table scope. This would > be a helpful metric. > I ran the WAL benchmark tool with 10 threads and didn't see any measurable > performance difference with the attached patch. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2385: HBASE-24950 Another solution for Splittable Meta
Apache-HBase commented on pull request #2385: URL: https://github.com/apache/hbase/pull/2385#issuecomment-696909317 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 36s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 14s | master passed | | +1 :green_heart: | compile | 3m 53s | master passed | | +1 :green_heart: | shadedjars | 7m 20s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 14s | master passed | | -0 :warning: | patch | 10m 37s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 4s | the patch passed | | +1 :green_heart: | compile | 3m 41s | the patch passed | | +1 :green_heart: | javac | 3m 41s | the patch passed | | +1 :green_heart: | shadedjars | 7m 19s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 12s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | javadoc | 0m 21s | hbase-common in the patch passed. | | +1 :green_heart: | javadoc | 0m 21s | hbase-client in the patch passed. | | +1 :green_heart: | javadoc | 0m 14s | hbase-zookeeper in the patch passed. | | +1 :green_heart: | javadoc | 0m 14s | hbase-balancer generated 0 new + 0 unchanged - 2 fixed = 0 total (was 2) | | -0 :warning: | javadoc | 0m 38s | hbase-server generated 2 new + 29 unchanged - 0 fixed = 31 total (was 29) | | +1 :green_heart: | javadoc | 0m 12s | hbase-it in the patch passed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 49s | hbase-protocol-shaded in the patch passed. | | -1 :x: | unit | 1m 57s | hbase-common in the patch failed. | | +1 :green_heart: | unit | 1m 18s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 0m 45s | hbase-zookeeper in the patch passed. | | +1 :green_heart: | unit | 0m 22s | hbase-balancer in the patch passed. | | -1 :x: | unit | 232m 5s | hbase-server in the patch failed. | | +1 :green_heart: | unit | 1m 6s | hbase-it in the patch passed. | | | | 279m 1s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2385/9/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2385 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 797c2ba96cde 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8a6299bb38 | | Default Java | 1.8.0_232 | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2385/9/artifact/yetus-jdk8-hadoop3-check/output/diff-javadoc-javadoc-hbase-server.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2385/9/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-common.txt | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2385/9/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2385/9/testReport/ | | Max. process+thread count | 2839 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-common hbase-client hbase-zookeeper hbase-balancer hbase-server hbase-it U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2385/9/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Inf
[GitHub] [hbase] Apache-HBase commented on pull request #2385: HBASE-24950 Another solution for Splittable Meta
Apache-HBase commented on pull request #2385: URL: https://github.com/apache/hbase/pull/2385#issuecomment-696905133 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 29s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 2s | master passed | | +1 :green_heart: | compile | 4m 43s | master passed | | +1 :green_heart: | shadedjars | 7m 40s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 17s | hbase-balancer in master failed. | | -0 :warning: | javadoc | 0m 27s | hbase-client in master failed. | | -0 :warning: | javadoc | 0m 17s | hbase-common in master failed. | | -0 :warning: | javadoc | 0m 45s | hbase-server in master failed. | | -0 :warning: | javadoc | 0m 19s | hbase-zookeeper in master failed. | | -0 :warning: | patch | 11m 14s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 45s | the patch passed | | +1 :green_heart: | compile | 4m 35s | the patch passed | | +1 :green_heart: | javac | 4m 35s | the patch passed | | +1 :green_heart: | shadedjars | 7m 37s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 20s | hbase-common in the patch failed. | | -0 :warning: | javadoc | 0m 24s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 17s | hbase-zookeeper in the patch failed. | | -0 :warning: | javadoc | 0m 18s | hbase-balancer in the patch failed. | | -0 :warning: | javadoc | 0m 51s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 6s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 2m 29s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 1m 38s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 0m 51s | hbase-zookeeper in the patch passed. | | +1 :green_heart: | unit | 0m 27s | hbase-balancer in the patch passed. | | +1 :green_heart: | unit | 218m 16s | hbase-server in the patch passed. | | +1 :green_heart: | unit | 1m 22s | hbase-it in the patch passed. | | | | 271m 8s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2385/9/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2385 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux e11cef819c9f 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8a6299bb38 | | Default Java | 2020-01-14 | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2385/9/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-balancer.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2385/9/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2385/9/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-common.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2385/9/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2385/9/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-zookeeper.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2385/9/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-common.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2385/9/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2385/9/artifact/y
[jira] [Commented] (HBASE-23887) BlockCache performance improve by reduce eviction rate
[
https://issues.apache.org/jira/browse/HBASE-23887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200283#comment-17200283
]
Josh Elser commented on HBASE-23887:
bq. While looking at the patch, I was wondering if we should implement this as
a separate Cache implementation that extends LruBlockCache, say
AdaptiveLruBlockCache or something? (restructure the code to override certain
methods like evict() etc). I think that helps with the following
While I want to find the time to come back and look some more at this, I think
Bharath's suggestion here is a good one. If you can tease this apart from
LruBlockCache, it will be easier to integrate this change. "Enabling" the
feature is a bit more sensical (e.g. we just use AdaptiveLruBlockCache instead
of changing the config property) and it removes (nearly?) all risk of changing
runtime semantics for users who don't want to use this.
I'll give a soft message of support for committing this to master, given the
analysis you've done (the google doc is great). I'll make a pass through to
make sure it all makes sense to me, but I don't see any big impediments if we
can decouple this from LruBlockCache.
ps: we _can_ leave the changes in LruBlockCache, but I think it will just
require more buy-in.
> BlockCache performance improve by reduce eviction rate
> --
>
> Key: HBASE-23887
> URL: https://issues.apache.org/jira/browse/HBASE-23887
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: Danil Lipovoy
>Assignee: Danil Lipovoy
>Priority: Minor
> Attachments: 1582787018434_rs_metrics.jpg,
> 1582801838065_rs_metrics_new.png, BC_LongRun.png,
> BlockCacheEvictionProcess.gif, BlockCacheEvictionProcess.gif, cmp.png,
> evict_BC100_vs_BC23.png, eviction_100p.png, eviction_100p.png,
> eviction_100p.png, gc_100p.png, graph.png, image-2020-06-07-08-11-11-929.png,
> image-2020-06-07-08-19-00-922.png, image-2020-06-07-12-07-24-903.png,
> image-2020-06-07-12-07-30-307.png, image-2020-06-08-17-38-45-159.png,
> image-2020-06-08-17-38-52-579.png, image-2020-06-08-18-35-48-366.png,
> image-2020-06-14-20-51-11-905.png, image-2020-06-22-05-57-45-578.png,
> ratio.png, ratio2.png, read_requests_100pBC_vs_23pBC.png, requests_100p.png,
> requests_100p.png, requests_new2_100p.png, requests_new_100p.png, scan.png,
> scan_and_gets.png, scan_and_gets2.png, wave.png
>
>
> Hi!
> I first time here, correct me please if something wrong.
> All latest information is here:
> [https://docs.google.com/document/d/1X8jVnK_3lp9ibpX6lnISf_He-6xrHZL0jQQ7hoTV0-g/edit?usp=sharing]
> I want propose how to improve performance when data in HFiles much more than
> BlockChache (usual story in BigData). The idea - caching only part of DATA
> blocks. It is good becouse LruBlockCache starts to work and save huge amount
> of GC.
> Sometimes we have more data than can fit into BlockCache and it is cause a
> high rate of evictions. In this case we can skip cache a block N and insted
> cache the N+1th block. Anyway we would evict N block quite soon and that why
> that skipping good for performance.
> ---
> Some information below isn't actual
> ---
>
>
> Example:
> Imagine we have little cache, just can fit only 1 block and we are trying to
> read 3 blocks with offsets:
> 124
> 198
> 223
> Current way - we put the block 124, then put 198, evict 124, put 223, evict
> 198. A lot of work (5 actions).
> With the feature - last few digits evenly distributed from 0 to 99. When we
> divide by modulus we got:
> 124 -> 24
> 198 -> 98
> 223 -> 23
> It helps to sort them. Some part, for example below 50 (if we set
> *hbase.lru.cache.data.block.percent* = 50) go into the cache. And skip
> others. It means we will not try to handle the block 198 and save CPU for
> other job. In the result - we put block 124, then put 223, evict 124 (3
> actions).
> See the picture in attachment with test below. Requests per second is higher,
> GC is lower.
>
> The key point of the code:
> Added the parameter: *hbase.lru.cache.data.block.percent* which by default =
> 100
>
> But if we set it 1-99, then will work the next logic:
>
>
> {code:java}
> public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean
> inMemory) {
> if (cacheDataBlockPercent != 100 && buf.getBlockType().isData())
> if (cacheKey.getOffset() % 100 >= cacheDataBlockPercent)
> return;
> ...
> // the same code as usual
> }
> {code}
>
> Other parameters help to control when this logic will be enabled. It means it
> will work only while heavy reading going on.
> hbase.lru.cache.heavy.eviction.count.limit - set how many times have to run
> eviction process that start to avoid of putting data to BlockCache
[GitHub] [hbase] Apache-HBase commented on pull request #2308: HBASE-20598 - Upgrade to JRuby 9.2
Apache-HBase commented on pull request #2308: URL: https://github.com/apache/hbase/pull/2308#issuecomment-696874037 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 46s | master passed | | +1 :green_heart: | compile | 2m 22s | master passed | | +1 :green_heart: | shadedjars | 6m 38s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 8s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 28s | the patch passed | | +1 :green_heart: | compile | 2m 23s | the patch passed | | +1 :green_heart: | javac | 2m 23s | the patch passed | | +1 :green_heart: | shadedjars | 6m 33s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 2m 5s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 286m 41s | root in the patch failed. | | | | 318m 58s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2308/8/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2308 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 0ed6b6ad5f84 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8a6299bb38 | | Default Java | 1.8.0_232 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2308/8/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2308/8/testReport/ | | Max. process+thread count | 4209 (vs. ulimit of 12500) | | modules | C: . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2308/8/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] joshelser commented on pull request #2308: HBASE-20598 - Upgrade to JRuby 9.2
joshelser commented on pull request #2308: URL: https://github.com/apache/hbase/pull/2308#issuecomment-696870760 Some subtle changes I'm noticing, the prompt is different. Now.. ``` hbase(#):003:0> ``` and before... ``` hbase(main):001:0> ``` Timestamps are also pretty-printed now: ``` ROW COLUMN+CELL r1 column=f:q, timestamp=2020-09-22T13:33:56.089, value=value 1 row(s) Took 0.1200 seconds ``` was before.. ``` hbase(main):001:0> scan 'j1' ROW COLUMN+CELL r1 column=f:q, timestamp=1600796036089, value=value 1 row(s) Took 0.3820 seconds ``` The timestamp one is probably something we should fix. I think the prompt fixing would be nice to fix, but I'm not sure how to do that without a deep dive :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-25067) Edit of log messages around async WAL Replication; checkstyle fixes; and a bugfix
[ https://issues.apache.org/jira/browse/HBASE-25067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-25067. --- Fix Version/s: 3.0.0-alpha-1 Hadoop Flags: Reviewed Resolution: Fixed Pushed to branch-2 and master, thanks for review [~zhangduo] > Edit of log messages around async WAL Replication; checkstyle fixes; and a > bugfix > - > > Key: HBASE-25067 > URL: https://issues.apache.org/jira/browse/HBASE-25067 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Edit of logging around region replicas: shortening and adding context. > Checkstyle fixes in edited files while I was in there. > Bug fix in AssignRegionHandler – was using M_RS_CLOSE_META to open > a Region instead of a M_RS_OPEN_META. > > Main reason for this issue is making the substantial adjacent issue > HBASE-25055 smaller in size/easier to review. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack closed pull request #2421: HBASE-25067 Edit of log messages around async WAL Replication; checks…
saintstack closed pull request #2421: URL: https://github.com/apache/hbase/pull/2421 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack commented on pull request #2421: HBASE-25067 Edit of log messages around async WAL Replication; checks…
saintstack commented on pull request #2421: URL: https://github.com/apache/hbase/pull/2421#issuecomment-696858663 Merged to branch-2 manually. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack merged pull request #2435: HBASE-25067 Edit of log messages around async WAL Replication; checks…
saintstack merged pull request #2435: URL: https://github.com/apache/hbase/pull/2435 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-25081) Up the container nproc uplimit to 30000
[ https://issues.apache.org/jira/browse/HBASE-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-25081. --- Fix Version/s: 3.0.0-alpha-1 Hadoop Flags: Reviewed Release Note: Ups the nproc (processes) limit from 12500 to 3 in yetus (so build container can have new limit). Assignee: Istvan Toth Resolution: Fixed > Up the container nproc uplimit to 3 > --- > > Key: HBASE-25081 > URL: https://issues.apache.org/jira/browse/HBASE-25081 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > Fix For: 3.0.0-alpha-1 > > > We (Apache Phoenix team) have recently switched our precommit tests to > Dockerized Yetus (mostly adopted from the solution in Hbase) > We see > java.lang.OutOfMemoryError: unable to create new native thread > errors , while Yetus shows > |Max. process+thread count|6833 (vs. ulimit of 12500)| > While I couldn't determine what was the job that we shared the Agent with at > the time, statistically it was very likely HBase, and an HBase job probably > failed with a similar error. > Some research has thrown up the official Docker docs: > [https://docs.docker.com/engine/reference/commandline/run/#set-ulimits-in-container-ulimit] > According to which it is not possible to set container level nprocs ulimit > with Docker. > All settings apply to the docker Daemon user instead, and the limit is shared > between all containers. > Based on this, I think that it makes no sense to set a container (really > docker user) nprocs ulimit any lower than the current hard limit of 3. > I have already set PROC_LIMIT=3 in the Phoenix Yetus personality, but it > is only a half solution until some Docker users set lower values, as the > later setting will apply as soon as the container is started. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25081) Up the container nproc uplimit to 30000
[ https://issues.apache.org/jira/browse/HBASE-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200193#comment-17200193 ] Michael Stack commented on HBASE-25081: --- No hurry [~stoty] Come back when you need it applied elsewhere. Resolving for now. Can open subtask when need it elsewhere. > Up the container nproc uplimit to 3 > --- > > Key: HBASE-25081 > URL: https://issues.apache.org/jira/browse/HBASE-25081 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Istvan Toth >Priority: Major > > We (Apache Phoenix team) have recently switched our precommit tests to > Dockerized Yetus (mostly adopted from the solution in Hbase) > We see > java.lang.OutOfMemoryError: unable to create new native thread > errors , while Yetus shows > |Max. process+thread count|6833 (vs. ulimit of 12500)| > While I couldn't determine what was the job that we shared the Agent with at > the time, statistically it was very likely HBase, and an HBase job probably > failed with a similar error. > Some research has thrown up the official Docker docs: > [https://docs.docker.com/engine/reference/commandline/run/#set-ulimits-in-container-ulimit] > According to which it is not possible to set container level nprocs ulimit > with Docker. > All settings apply to the docker Daemon user instead, and the limit is shared > between all containers. > Based on this, I think that it makes no sense to set a container (really > docker user) nprocs ulimit any lower than the current hard limit of 3. > I have already set PROC_LIMIT=3 in the Phoenix Yetus personality, but it > is only a half solution until some Docker users set lower values, as the > later setting will apply as soon as the container is started. -- This message was sent by Atlassian Jira (v8.3.4#803005)
