[GitHub] [hbase] anoopsjohn commented on a change in pull request #3712: HBASE-26320 Implement a separate thread pool for the LogCleaner
anoopsjohn commented on a change in pull request #3712:
URL: https://github.com/apache/hbase/pull/3712#discussion_r721030034
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
##
@@ -64,6 +64,8 @@
*/
public static final String CHORE_POOL_SIZE =
"hbase.cleaner.scan.dir.concurrent.size";
static final String DEFAULT_CHORE_POOL_SIZE = "0.25";
+ public static final String LOG_CLEANER_CHORE_SIZE =
"hbase.log.cleaner.scan.dir.concurrent.size";
Review comment:
So now we will have (by default) a pool of 1 thread for scan and
cleaning of Archived HFiles.
For old WALs the thread pool's default size will be 25% of #cores
Can we add some code level comments here?
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/DirScanPool.java
##
@@ -109,11 +117,21 @@ synchronized void tryUpdatePoolSize(long timeout) {
break;
}
}
-LOG.info("Update chore's pool size from {} to {}", pool.getPoolSize(),
size);
+LOG.info("Update {} chore's pool size from {} to {}", name,
pool.getPoolSize(), size);
pool.setCorePoolSize(size);
}
public int getSize() {
return size;
}
+
+ public static DirScanPool getArchiveScanner(Configuration conf) {
Review comment:
+1. Got the same Q when read the method name at begin of patch.
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/DirScanPool.java
##
@@ -39,21 +39,28 @@
private final ThreadPoolExecutor pool;
private int cleanerLatch;
private boolean reconfigNotification;
+ private final String cleanerPoolConfig;
+ private final String cleanerPoolConfigDefault;
+ private final String name;
- public DirScanPool(Configuration conf) {
-String poolSize = conf.get(CleanerChore.CHORE_POOL_SIZE,
CleanerChore.DEFAULT_CHORE_POOL_SIZE);
+ private DirScanPool(Configuration conf, String cleanerPoolConfig, String
cleanerPoolConfigDefault,
Review comment:
+1 for this suggested name. Looks clean.
Can we think of making this more clean?
DirScanPool can have a private Enum Type (for ArchivedHFilesCleaner,
OldLogsCleaner)
The constructor can only take the type. So the other details of name,
configName, configDefault are considered based on type. Infact type can have a
method to return the poolSize.
We dont have to even store the cleanerPoolConfigDefault
I think its used only on configUpdate method. That can use
Configuration#get(String) and have a null check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-24601) Change default Hfile storage policy from HOT to NONE for HDFS
[
https://issues.apache.org/jira/browse/HBASE-24601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17423758#comment-17423758
]
Anoop Sam John commented on HBASE-24601:
Better we can mark this as an incompatible change? This changes a behaviour
even though we call old behave as bug or not.
> Change default Hfile storage policy from HOT to NONE for HDFS
> -
>
> Key: HBASE-24601
> URL: https://issues.apache.org/jira/browse/HBASE-24601
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 2.2.5
>Reporter: dark_num
>Assignee: dark_num
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2
>
> Attachments: hfile-policy-problem.png
>
>
> The current HBase hfile directory specifies the *HOT* storage strategy by
> default(`hbase.hstore.blockingStoreFiles` ), which is different from the
> default value of HDFS *heterogeneous* strategy now(should be empty)
> It should be set to *NONE* like `hbase.wal.storage.policy`'s default value on
> HBASE-20691, and the reason for modification is the same
> Otherwise, it could have a significant impact because most HDFS Managers
> doesn't know the default value has been changed to *HOT* by hbase, for
> example:
> HDFS manger want to set hfile's dir to ONE_SSD for speed up read, they will
> do:
> {code:java}
> // Set Hbas Hfile dir to ONE_SSD for speed up read
> // The policy that expects all subdirectories to inherit the parent directory
> by default (empty by default)
> bin/hdfs storagepolicies -setStoragePolicy -path /home/hbase/data -policy
> ONE_SSD
> bin/hdfs storagepolicies -setStoragePolicy -path /home/hbase/WALs -policy
> ALL_SSD
> // If we do not change Hbase default setting, we will find this
> bin/hdfs storagepolicies -getStoragePolicy -path
> /home/hbase/data/hbase/meta//info
> return "HOT" instead of "ONE_SSD"{code}
> And here is a pic to show the problem ↑:
> !hfile-policy-problem.png!
> However, if Hfile uses the *HOT* strategy by default, then we will find that
> the actually stored directory has been set to HOT, then it will *not inherit*
> the *ONE_SSD* of the parent directory policy, and if we set *NONE* in hbase,
> this is problem can be avoided
> Unless someone sets other values manually, then we believe the operator
> knows what he is doing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3720: HBASE-26325 Backport the HBASE-26305 to branch-1. Move NavigableSet add operation to writer thread in BucketCache
Apache-HBase commented on pull request #3720: URL: https://github.com/apache/hbase/pull/3720#issuecomment-933104180 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | docker | 5m 34s | Docker failed to build yetus/hbase:633d966bfe. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hbase/pull/3720 | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3720/1/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] YutSean opened a new pull request #3720: HBASE-26325 Backport the HBASE-26305 to branch-1. Move NavigableSet add operation to writer thread in BucketCache
YutSean opened a new pull request #3720: URL: https://github.com/apache/hbase/pull/3720 https://issues.apache.org/jira/browse/HBASE-26325 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3719: HBASE-26325 Backport HBASE-26305 to branch-1. Move NavigableSet add operation to writer thread in BucketCache
Apache-HBase commented on pull request #3719: URL: https://github.com/apache/hbase/pull/3719#issuecomment-933101484 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | docker | 10m 40s | Docker failed to build yetus/hbase:633d966bfe. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hbase/pull/3719 | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3719/1/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] YutSean closed pull request #3719: HBASE-26325 Backport HBASE-26305 to branch-1. Move NavigableSet add operation to writer thread in BucketCache
YutSean closed pull request #3719: URL: https://github.com/apache/hbase/pull/3719 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] YutSean opened a new pull request #3719: HBASE-26325 Backport HBASE-26305 to branch-1. Move NavigableSet add operation to writer thread in BucketCache
YutSean opened a new pull request #3719: URL: https://github.com/apache/hbase/pull/3719 https://issues.apache.org/jira/browse/HBASE-26325 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3718: Backport the HBASE-26305 to branch-1. Move NavigableSet add operation to writer thread in BucketCache
Apache-HBase commented on pull request #3718: URL: https://github.com/apache/hbase/pull/3718#issuecomment-933097665 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 0s | Docker mode activated. | | -1 :x: | patch | 0m 8s | https://github.com/apache/hbase/pull/3718 does not apply to branch-1. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hbase/pull/3718 | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3718/1/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] YutSean closed pull request #3718: Backport the HBASE-26305 to branch-1. Move NavigableSet add operation to writer thread in BucketCache
YutSean closed pull request #3718: URL: https://github.com/apache/hbase/pull/3718 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] YutSean opened a new pull request #3718: Backport the HBASE-26305 to branch-1. Move NavigableSet add operation to writer thread in BucketCache
YutSean opened a new pull request #3718: URL: https://github.com/apache/hbase/pull/3718 https://issues.apache.org/jira/browse/HBASE-26325 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-26325) Backport HBASE-26305 to branch-1
Yutong Xiao created HBASE-26325: --- Summary: Backport HBASE-26305 to branch-1 Key: HBASE-26325 URL: https://issues.apache.org/jira/browse/HBASE-26325 Project: HBase Issue Type: Improvement Reporter: Yutong Xiao Assignee: Yutong Xiao Backport the HBASE-26305 to branch-1. Move NavigableSet add operation to writer thread in BucketCache -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3691: HBASE-26259 Fallback support to pure Java compression
Apache-HBase commented on pull request #3691: URL: https://github.com/apache/hbase/pull/3691#issuecomment-933091343 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 32s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 39s | master passed | | +1 :green_heart: | compile | 8m 24s | master passed | | +1 :green_heart: | checkstyle | 1m 58s | master passed | | +1 :green_heart: | spotbugs | 13m 11s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 48s | the patch passed | | +1 :green_heart: | compile | 8m 55s | the patch passed | | +1 :green_heart: | javac | 8m 55s | the patch passed | | -0 :warning: | checkstyle | 2m 2s | root: The patch generated 55 new + 73 unchanged - 15 fixed = 128 total (was 88) | | +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 12s | The patch has no ill-formed XML file. | | +1 :green_heart: | hadoopcheck | 19m 21s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | spotbugs | 18m 35s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 2m 34s | The patch does not generate ASF License warnings. | | | | 94m 2s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3691/19/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3691 | | Optional Tests | dupname asflicense javac hadoopcheck xml compile spotbugs hbaseanti checkstyle | | uname | Linux 8af214af5692 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f65b769e8c | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3691/19/artifact/yetus-general-check/output/diff-checkstyle-root.txt | | Max. process+thread count | 141 (vs. ulimit of 3) | | modules | C: hbase-common hbase-resource-bundle hbase-server hbase-mapreduce hbase-compression hbase-compression/hbase-compression-aircompressor hbase-compression/hbase-compression-lz4 hbase-compression/hbase-compression-snappy hbase-compression/hbase-compression-xz hbase-compression/hbase-compression-zstd hbase-assembly . U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3691/19/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] apurtell commented on pull request #3691: HBASE-26259 Fallback support to pure Java compression
apurtell commented on pull request #3691: URL: https://github.com/apache/hbase/pull/3691#issuecomment-933062764 Let's try this again... Pushed an update for the spotbugs warnings, after rebasing on current master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase removed a comment on pull request #3691: HBASE-26259 Fallback support to pure Java compression
Apache-HBase removed a comment on pull request #3691: URL: https://github.com/apache/hbase/pull/3691#issuecomment-932703082 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-26324) Reuse compressors and decompressors in WAL CompressionContext
Andrew Kyle Purtell created HBASE-26324: --- Summary: Reuse compressors and decompressors in WAL CompressionContext Key: HBASE-26324 URL: https://issues.apache.org/jira/browse/HBASE-26324 Project: HBase Issue Type: Sub-task Components: wal Reporter: Andrew Kyle Purtell Assignee: Andrew Kyle Purtell Fix For: 2.5.0, 3.0.0-alpha-2 WAL value compression CompressionContext allocates more compressors and decompressors than it requires and does not attempt to return those instances to their respective pools. Found while profiling. Harmless, but let's fix it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3716: HBASE-26323 introduce a SnapshotProcedure
Apache-HBase commented on pull request #3716: URL: https://github.com/apache/hbase/pull/3716#issuecomment-932996683 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 58s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 2s | master passed | | +1 :green_heart: | compile | 2m 52s | master passed | | +1 :green_heart: | shadedjars | 9m 13s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 21s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 51s | the patch passed | | +1 :green_heart: | compile | 2m 53s | the patch passed | | +1 :green_heart: | javac | 2m 53s | the patch passed | | +1 :green_heart: | shadedjars | 9m 11s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 19s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 3s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 1m 43s | hbase-client in the patch passed. | | -1 :x: | unit | 206m 44s | hbase-server in the patch failed. | | | | 250m 0s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3716/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3716 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 41f7283ebf47 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f65b769e8c | | Default Java | AdoptOpenJDK-11.0.10+9 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3716/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3716/1/testReport/ | | Max. process+thread count | 2919 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3716/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3716: HBASE-26323 introduce a SnapshotProcedure
Apache-HBase commented on pull request #3716: URL: https://github.com/apache/hbase/pull/3716#issuecomment-932986399 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 17s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 55s | master passed | | +1 :green_heart: | compile | 2m 15s | master passed | | +1 :green_heart: | shadedjars | 8m 19s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 15s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 45s | the patch passed | | +1 :green_heart: | compile | 2m 13s | the patch passed | | +1 :green_heart: | javac | 2m 13s | the patch passed | | +1 :green_heart: | shadedjars | 8m 13s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 13s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 47s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 1m 18s | hbase-client in the patch passed. | | -1 :x: | unit | 149m 34s | hbase-server in the patch failed. | | | | 186m 38s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3716/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3716 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 295faa7231f2 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f65b769e8c | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | unit | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3716/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3716/1/testReport/ | | Max. process+thread count | 4712 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3716/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3716: HBASE-26323 introduce a SnapshotProcedure
Apache-HBase commented on pull request #3716: URL: https://github.com/apache/hbase/pull/3716#issuecomment-932966834 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 25s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 57s | master passed | | +1 :green_heart: | compile | 5m 34s | master passed | | +1 :green_heart: | checkstyle | 1m 50s | master passed | | +1 :green_heart: | spotbugs | 7m 3s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 13s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 39s | the patch passed | | +1 :green_heart: | compile | 5m 35s | the patch passed | | +1 :green_heart: | cc | 5m 35s | the patch passed | | +1 :green_heart: | javac | 5m 35s | the patch passed | | -0 :warning: | checkstyle | 0m 31s | hbase-client: The patch generated 1 new + 186 unchanged - 0 fixed = 187 total (was 186) | | -0 :warning: | checkstyle | 1m 7s | hbase-server: The patch generated 23 new + 157 unchanged - 1 fixed = 180 total (was 158) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 18m 32s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0. | | +1 :green_heart: | hbaseprotoc | 2m 4s | the patch passed | | -1 :x: | spotbugs | 2m 20s | hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 38s | The patch does not generate ASF License warnings. | | | | 68m 3s | | | Reason | Tests | |---:|:--| | FindBugs | module:hbase-server | | | Inconsistent synchronization of org.apache.hadoop.hbase.master.procedure.ServerRemoteProcedure.dispatched; locked 60% of time Unsynchronized access at SnapshotVerifyProcedure.java:60% of time Unsynchronized access at SnapshotVerifyProcedure.java:[line 97] | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3716/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3716 | | Optional Tests | dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile cc hbaseprotoc prototool | | uname | Linux 4321f64144be 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f65b769e8c | | Default Java | AdoptOpenJDK-1.8.0_282-b08 | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3716/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-client.txt | | checkstyle | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3716/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | spotbugs | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3716/1/artifact/yetus-general-check/output/new-spotbugs-hbase-server.html | | Max. process+thread count | 96 (vs. ulimit of 3) | | modules | C: hbase-protocol-shaded hbase-client hbase-server U: . | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3716/1/console | | versions | git=2.17.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-26278) Add preRequestCompaction method for AccessController
[ https://issues.apache.org/jira/browse/HBASE-26278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yulin Niu resolved HBASE-26278. --- Resolution: Fixed > Add preRequestCompaction method for AccessController > > > Key: HBASE-26278 > URL: https://issues.apache.org/jira/browse/HBASE-26278 > Project: HBase > Issue Type: Sub-task >Reporter: Yulin Niu >Assignee: Yulin Niu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-26278) Add preRequestCompaction method for AccessController
[ https://issues.apache.org/jira/browse/HBASE-26278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17423669#comment-17423669 ] Yulin Niu commented on HBASE-26278: --- push to branch HBASE-25714, thanks [~zhangduo]'s reviewing > Add preRequestCompaction method for AccessController > > > Key: HBASE-26278 > URL: https://issues.apache.org/jira/browse/HBASE-26278 > Project: HBase > Issue Type: Sub-task >Reporter: Yulin Niu >Assignee: Yulin Niu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] nyl3532016 merged pull request #3677: HBASE-26278 Add preRequestCompaction method for AccessController
nyl3532016 merged pull request #3677: URL: https://github.com/apache/hbase/pull/3677 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-26323) introduce a SnapshotProcedure
[ https://issues.apache.org/jira/browse/HBASE-26323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17423668#comment-17423668 ] ruanhui commented on HBASE-26323: - sorry. I don't know why the pull request didn't link, can I just put it here ? https://github.com/apache/hbase/pull/3716 > introduce a SnapshotProcedure > - > > Key: HBASE-26323 > URL: https://issues.apache.org/jira/browse/HBASE-26323 > Project: HBase > Issue Type: New Feature > Components: proc-v2, snapshots >Reporter: ruanhui >Assignee: ruanhui >Priority: Minor > > Currently，snapshot in hbase uses zk as coordinator. It has some limitations, > a. Snapshot maybe fails when there are region server crashes. > b. Snapshot maybe failed when master restarts. > c. Only one snapshot per table can be taken in a time. > d. Snapshot verify will be handled by master, which may take long time when > our table has a large number of regions, for example 1. > > Since we have procedure v2 framework now, it is possible to solve the above > problems. So here is a procedure2-based snapshot implementation. It has some > goals, > a. Snapshot can continue when there are region server crashes. > b. Snapshot can continue when master restarts. > c. More than one snapshot per table can be taken in a time. > d. We can use region servers to verify snapshot to accelerate procedure. > > Here are some details about implementation. > *SnapshotProcedure* > SnapshotProcedure is used to take snapshot on a table. It acquires shared > table lock on the snapshot table and hold the shared lock during suspend and > yield. > *SnapshotRegionProcedure* > SnapshotRegionProcedure is used to take snapshot on a specific region of the > snapshot table. It acquires exclusive region lock and releases lock during > suspend and yield. Before dispatch remote snapshot operations to region > server, it will check target region in RIT or not. If target region is in > RIT, it will sleep some time and retry. > *SnapshotVerifyProcedure* > SnapshotVerifyProcedure is used to send snapshot verify request to region > server. If snapshot is corrupted, it will notify parent snapshot to retry. > When remote region server is crashed, it will choose another online server > and retry. > > I would be very grateful for any advice and guidance. Is anyone interested in > taking a look? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #3717: HBASE-26318 Publish test logs for flaky jobs to nightlies
Apache-HBase commented on pull request #3717: URL: https://github.com/apache/hbase/pull/3717#issuecomment-932958133 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 56s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for branch | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 6s | Maven dependency ordering for patch | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | ||| _ Other Tests _ | | +0 :ok: | asflicense | 0m 0s | ASF License check generated no output? | | | | 5m 25s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3717/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/3717 | | Optional Tests | dupname asflicense | | uname | Linux df4dd9b7ebf3 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / f65b769e8c | | Max. process+thread count | 46 (vs. ulimit of 3) | | modules | C: U: | | Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3717/1/console | | versions | git=2.17.1 maven=3.6.3 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #3717: HBASE-26318 Publish test logs for flaky jobs to nightlies
Apache-HBase commented on pull request #3717: URL: https://github.com/apache/hbase/pull/3717#issuecomment-932957554 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #3717: HBASE-26318 Publish test logs for flaky jobs to nightlies
Apache9 commented on pull request #3717: URL: https://github.com/apache/hbase/pull/3717#issuecomment-932956997 Please see this build https://ci-hadoop.apache.org/job/HBase/job/HBase-Flaky-Tests/job/HBASE-26318/10/ The artifacts url is: https://nightlies.apache.org/hbase/HBase/HBase-Flaky-Tests/HBASE-26318/10/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HBASE-26323) introduce a SnapshotProcedure
[ https://issues.apache.org/jira/browse/HBASE-26323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ruanhui reassigned HBASE-26323: --- Assignee: ruanhui > introduce a SnapshotProcedure > - > > Key: HBASE-26323 > URL: https://issues.apache.org/jira/browse/HBASE-26323 > Project: HBase > Issue Type: New Feature > Components: proc-v2, snapshots >Reporter: ruanhui >Assignee: ruanhui >Priority: Minor > > Currently，snapshot in hbase uses zk as coordinator. It has some limitations, > a. Snapshot maybe fails when there are region server crashes. > b. Snapshot maybe failed when master restarts. > c. Only one snapshot per table can be taken in a time. > d. Snapshot verify will be handled by master, which may take long time when > our table has a large number of regions, for example 1. > > Since we have procedure v2 framework now, it is possible to solve the above > problems. So here is a procedure2-based snapshot implementation. It has some > goals, > a. Snapshot can continue when there are region server crashes. > b. Snapshot can continue when master restarts. > c. More than one snapshot per table can be taken in a time. > d. We can use region servers to verify snapshot to accelerate procedure. > > Here are some details about implementation. > *SnapshotProcedure* > SnapshotProcedure is used to take snapshot on a table. It acquires shared > table lock on the snapshot table and hold the shared lock during suspend and > yield. > *SnapshotRegionProcedure* > SnapshotRegionProcedure is used to take snapshot on a specific region of the > snapshot table. It acquires exclusive region lock and releases lock during > suspend and yield. Before dispatch remote snapshot operations to region > server, it will check target region in RIT or not. If target region is in > RIT, it will sleep some time and retry. > *SnapshotVerifyProcedure* > SnapshotVerifyProcedure is used to send snapshot verify request to region > server. If snapshot is corrupted, it will notify parent snapshot to retry. > When remote region server is crashed, it will choose another online server > and retry. > > I would be very grateful for any advice and guidance. Is anyone interested in > taking a look? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26323) introduce a SnapshotProcedure
ruanhui created HBASE-26323: --- Summary: introduce a SnapshotProcedure Key: HBASE-26323 URL: https://issues.apache.org/jira/browse/HBASE-26323 Project: HBase Issue Type: New Feature Components: proc-v2, snapshots Reporter: ruanhui Currently，snapshot in hbase uses zk as coordinator. It has some limitations, a. Snapshot maybe fails when there are region server crashes. b. Snapshot maybe failed when master restarts. c. Only one snapshot per table can be taken in a time. d. Snapshot verify will be handled by master, which may take long time when our table has a large number of regions, for example 1. Since we have procedure v2 framework now, it is possible to solve the above problems. So here is a procedure2-based snapshot implementation. It has some goals, a. Snapshot can continue when there are region server crashes. b. Snapshot can continue when master restarts. c. More than one snapshot per table can be taken in a time. d. We can use region servers to verify snapshot to accelerate procedure. Here are some details about implementation. *SnapshotProcedure* SnapshotProcedure is used to take snapshot on a table. It acquires shared table lock on the snapshot table and hold the shared lock during suspend and yield. *SnapshotRegionProcedure* SnapshotRegionProcedure is used to take snapshot on a specific region of the snapshot table. It acquires exclusive region lock and releases lock during suspend and yield. Before dispatch remote snapshot operations to region server, it will check target region in RIT or not. If target region is in RIT, it will sleep some time and retry. *SnapshotVerifyProcedure* SnapshotVerifyProcedure is used to send snapshot verify request to region server. If snapshot is corrupted, it will notify parent snapshot to retry. When remote region server is crashed, it will choose another online server and retry. I would be very grateful for any advice and guidance. Is anyone interested in taking a look? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-26317) Publish the test logs for pre commit jenkins job to nightlies
[ https://issues.apache.org/jira/browse/HBASE-26317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-26317: -- Status: Patch Available (was: In Progress) > Publish the test logs for pre commit jenkins job to nightlies > -- > > Key: HBASE-26317 > URL: https://issues.apache.org/jira/browse/HBASE-26317 > Project: HBase > Issue Type: Sub-task > Components: jenkins, scripts >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-26313) Publish the test logs for our nightly jobs to nightlies.apache.org
[ https://issues.apache.org/jira/browse/HBASE-26313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-26313: -- Status: Patch Available (was: In Progress) > Publish the test logs for our nightly jobs to nightlies.apache.org > -- > > Key: HBASE-26313 > URL: https://issues.apache.org/jira/browse/HBASE-26313 > Project: HBase > Issue Type: Sub-task > Components: jenkins, scripts >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > To save spaces on jenkins build machine. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 commented on pull request #3644: Clean shared state pollution to avoid flaky tests.
Apache9 commented on pull request #3644: URL: https://github.com/apache/hbase/pull/3644#issuecomment-932945210 HBase uses JIRA to track issues, so please open a jira first at https://issues.apache.org/jira? And on the patch, I think we'd better add the in a tearDown method? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@hbase.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
