[jira] [Reopened] (HBASE-25213) Should request Compaction when bulkLoadHFiles is done
[ https://issues.apache.org/jira/browse/HBASE-25213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin reopened HBASE-25213: -- UT: TestLowLatencySpaceQuotas#testBulkLoading will check the space used must bigger than or equal to the files loaded. After this pr, it occasionally fail. So,should disable compaction during this ut > Should request Compaction when bulkLoadHFiles is done > - > > Key: HBASE-25213 > URL: https://issues.apache.org/jira/browse/HBASE-25213 > Project: HBase > Issue Type: Improvement >Reporter: niuyulin >Assignee: niuyulin >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.4.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25281) Bulkload split hfile too many times due to unreasonable split point
[
https://issues.apache.org/jira/browse/HBASE-25281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niuyulin updated HBASE-25281:
-
Description:
[https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/BulkLoadHFilesTool.java#L688]
if hfile span multi regions, for example A,B,C,D,E,F(the start key of these
regions are in ascending order), we should use region C endkey to split, not
region A. In this way, we can get equal .top and .bottom hfiles, reduce time
complexity of split from O( n ) to O(logn),decrease invoke of {{bulkLoad}} rpc
to regionserver ,and also avoid write amplification during {{copyHFileHalf}}
was:
[https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/BulkLoadHFilesTool.java#L688]
if hfile span multi regions, for example A,B,C,D,E,F(the start key of these
regions are in ascending order), we should use region C endkey to split, not
region A. In this way, we can get equal .top and .bottom hfiles, reduce time
complexity of split from O(n) to O(logn),decrease invoke of {{bulkLoad}} rpc to
regionserver ,and also avoid write amplification during {{copyHFileHalf}}
> Bulkload split hfile too many times due to unreasonable split point
> ---
>
> Key: HBASE-25281
> URL: https://issues.apache.org/jira/browse/HBASE-25281
> Project: HBase
> Issue Type: Improvement
> Components: tooling
>Reporter: niuyulin
>Assignee: niuyulin
>Priority: Minor
> Attachments: bulkload.png
>
>
> [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/BulkLoadHFilesTool.java#L688]
> if hfile span multi regions, for example A,B,C,D,E,F(the start key of these
> regions are in ascending order), we should use region C endkey to split, not
> region A. In this way, we can get equal .top and .bottom hfiles, reduce time
> complexity of split from O( n ) to O(logn),decrease invoke of {{bulkLoad}}
> rpc to regionserver ,and also avoid write amplification during
> {{copyHFileHalf}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25281) Bulkload split hfile too many times due to unreasonable split point
[
https://issues.apache.org/jira/browse/HBASE-25281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niuyulin updated HBASE-25281:
-
Description:
[https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/BulkLoadHFilesTool.java#L688]
if hfile span multi regions, for example A,B,C,D,E,F(the start key of these
regions are in ascending order), we should use region C endkey to split, not
region A. In this way, we can get equal .top and .bottom hfiles, reduce time
complexity of split from O(n) to O(logn),decrease invoke of {{bulkLoad}} rpc to
regionserver ,and also avoid write amplification during {{copyHFileHalf}}
was:
[https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/BulkLoadHFilesTool.java#L688]
the split point is not in middle, If the hfile span N regions, we need split
N-1 times now
> Bulkload split hfile too many times due to unreasonable split point
> ---
>
> Key: HBASE-25281
> URL: https://issues.apache.org/jira/browse/HBASE-25281
> Project: HBase
> Issue Type: Improvement
> Components: tooling
>Reporter: niuyulin
>Assignee: niuyulin
>Priority: Minor
> Attachments: bulkload.png
>
>
> [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/BulkLoadHFilesTool.java#L688]
> if hfile span multi regions, for example A,B,C,D,E,F(the start key of these
> regions are in ascending order), we should use region C endkey to split, not
> region A. In this way, we can get equal .top and .bottom hfiles, reduce time
> complexity of split from O(n) to O(logn),decrease invoke of {{bulkLoad}} rpc
> to regionserver ,and also avoid write amplification during {{copyHFileHalf}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25281) Bulkload split hfile too many times due to unreasonable split point
[
https://issues.apache.org/jira/browse/HBASE-25281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niuyulin updated HBASE-25281:
-
Description:
[https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/BulkLoadHFilesTool.java#L688]
if hfile span multi regions, for example A,B,C,D,E,F(the start key of these
regions are in ascending order), we should use region C endkey to split, not
region A. In this way, we can get equal .top and .bottom hfiles, reduce time
complexity of split from O(n) to O(logn),decrease invoke of {{bulkLoad}} rpc to
regionserver ,and also avoid write amplification during {{copyHFileHalf}}
was:
[https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/BulkLoadHFilesTool.java#L688]
if hfile span multi regions, for example A,B,C,D,E,F(the start key of these
regions are in ascending order), we should use region C endkey to split, not
region A. In this way, we can get equal .top and .bottom hfiles, reduce time
complexity of split from O(n) to O(logn),decrease invoke of {{bulkLoad}} rpc to
regionserver ,and also avoid write amplification during {{copyHFileHalf}}
> Bulkload split hfile too many times due to unreasonable split point
> ---
>
> Key: HBASE-25281
> URL: https://issues.apache.org/jira/browse/HBASE-25281
> Project: HBase
> Issue Type: Improvement
> Components: tooling
>Reporter: niuyulin
>Assignee: niuyulin
>Priority: Minor
> Attachments: bulkload.png
>
>
> [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/BulkLoadHFilesTool.java#L688]
> if hfile span multi regions, for example A,B,C,D,E,F(the start key of these
> regions are in ascending order), we should use region C endkey to split, not
> region A. In this way, we can get equal .top and .bottom hfiles, reduce time
> complexity of split from O(n) to O(logn),decrease invoke of {{bulkLoad}} rpc
> to regionserver ,and also avoid write amplification during {{copyHFileHalf}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25213) Should request Compaction when bulkLoadHFiles is done
[ https://issues.apache.org/jira/browse/HBASE-25213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236642#comment-17236642 ] niuyulin commented on HBASE-25213: -- of course, I have submitted another PR for branch-2 [~zghao] > Should request Compaction when bulkLoadHFiles is done > - > > Key: HBASE-25213 > URL: https://issues.apache.org/jira/browse/HBASE-25213 > Project: HBase > Issue Type: Improvement >Reporter: niuyulin >Assignee: niuyulin >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25281) Bulkload split hfile too many times due to unreasonable split point
[ https://issues.apache.org/jira/browse/HBASE-25281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin updated HBASE-25281: - Component/s: tooling > Bulkload split hfile too many times due to unreasonable split point > --- > > Key: HBASE-25281 > URL: https://issues.apache.org/jira/browse/HBASE-25281 > Project: HBase > Issue Type: Improvement > Components: tooling >Reporter: niuyulin >Assignee: niuyulin >Priority: Minor > Attachments: bulkload.png > > > [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/BulkLoadHFilesTool.java#L688] > the split point is not in middle, If the hfile span N regions, we need split > N-1 times now -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25281) Bulkload split hfile too many times due to unreasonable split point
[ https://issues.apache.org/jira/browse/HBASE-25281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin updated HBASE-25281: - Attachment: bulkload.png > Bulkload split hfile too many times due to unreasonable split point > --- > > Key: HBASE-25281 > URL: https://issues.apache.org/jira/browse/HBASE-25281 > Project: HBase > Issue Type: Improvement >Reporter: niuyulin >Assignee: niuyulin >Priority: Minor > Attachments: bulkload.png > > > [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/BulkLoadHFilesTool.java#L688] > the split point is not in middle, If the hfile span N regions, we need split > N-1 times now -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25281) Bulkload split hfile too many times due to unreasonable split point
niuyulin created HBASE-25281: Summary: Bulkload split hfile too many times due to unreasonable split point Key: HBASE-25281 URL: https://issues.apache.org/jira/browse/HBASE-25281 Project: HBase Issue Type: Improvement Reporter: niuyulin Assignee: niuyulin [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/BulkLoadHFilesTool.java#L688] the split point is not in middle, If the hfile span N regions, we need split N-1 times now -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-25276) Need to throw the original exception in HRegion#openHRegion
[
https://issues.apache.org/jira/browse/HBASE-25276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niuyulin reassigned HBASE-25276:
Assignee: niuyulin
> Need to throw the original exception in HRegion#openHRegion
> ---
>
> Key: HBASE-25276
> URL: https://issues.apache.org/jira/browse/HBASE-25276
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: niuyulin
>Priority: Major
>
> {code:java}
> protected HRegion openHRegion(final CancelableProgressable reporter)
> throws IOException {
> try {
> ..
> } catch(Throwable t) {
> // By coprocessor path wrong region will open failed,
> // MetricsRegionWrapperImpl is already init and not close,
> // add region close when open failed
> this.close(); // This may throw IOException too.
> throw t;
> }
> return this;
> }
> {code}
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java#L8144
> this.close() may throw IOException too. The original exception will not be
> recorded.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25213) Should request Compaction when bulkLoadHFiles is done
[ https://issues.apache.org/jira/browse/HBASE-25213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin updated HBASE-25213: - Summary: Should request Compaction when bulkLoadHFiles is done (was: Should request Compaction after bulkLoadHFiles is done) > Should request Compaction when bulkLoadHFiles is done > - > > Key: HBASE-25213 > URL: https://issues.apache.org/jira/browse/HBASE-25213 > Project: HBase > Issue Type: Improvement >Reporter: niuyulin >Assignee: niuyulin >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25116) RegionMonitor support RegionTask count normalize
[ https://issues.apache.org/jira/browse/HBASE-25116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17228334#comment-17228334 ] niuyulin commented on HBASE-25116: -- What is about the canary access that slows user-requests? =>the meta access may slow user-requests Should these options be made into command-line options for the Canary tool rather than internal configs? =>yes,made into command-line options is more friendly to users A task maps to a Region? Would it help if we talked of sampling rather than task counts? => Yes, a task per region, but I think sampling is more complicated than task-count. normally the read task(user region) is 1w+ and write task(canary region) is 1k- maybe 100 +, so we set task-count-max: 5k , task-count-min:2k, and read task will be sampling and write task will be enlarged. The task normalize will work only when task count is not in task count range we set > RegionMonitor support RegionTask count normalize > > > Key: HBASE-25116 > URL: https://issues.apache.org/jira/browse/HBASE-25116 > Project: HBase > Issue Type: Improvement >Reporter: niuyulin >Assignee: niuyulin >Priority: Minor > > large count of region task from canary may affect user normal request, > meanwhile if region task is few, the availability monitoring may shake for > occasional exception. > so , if the task count is large , we will randomly trim tasks for each table, > according to the raito of the table region count in whole tasks region count. > If the task count is few, we will repeat tasks -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25053) WAL replay should ignore 0-length files
[
https://issues.apache.org/jira/browse/HBASE-25053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17225808#comment-17225808
]
niuyulin commented on HBASE-25053:
--
[~stack] mind help review this pr and see if this issue can be closed
> WAL replay should ignore 0-length files
> ---
>
> Key: HBASE-25053
> URL: https://issues.apache.org/jira/browse/HBASE-25053
> Project: HBase
> Issue Type: Bug
> Components: master, regionserver
>Affects Versions: 2.3.1
>Reporter: Nick Dimiduk
>Assignee: niuyulin
>Priority: Major
>
> I overdrove a small testing cluster, filling HDFS. After cleaning up data to
> bring HBase back up, I noticed all masters -refused to start- abort. Logs
> complain of seeking past EOF. Indeed the last wal file name logged is a
> 0-length file. WAL replay should gracefully skip and clean up such an empty
> file.
> {noformat}
> 2020-09-16 19:51:30,297 ERROR org.apache.hadoop.hbase.master.HMaster: Failed
> to become active master
> java.io.EOFException: Cannot seek after EOF
> at
> org.apache.hadoop.hdfs.DFSInputStream.seek(DFSInputStream.java:1448)
> at
> org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:66)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initInternal(ProtobufLogReader.java:211)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initReader(ProtobufLogReader.java:173)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:64)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:168)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:323)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:305)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:293)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:429)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:4859)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:4765)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:1014)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:956)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7496)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegionFromTableDir(HRegion.java:7454)
> at
> org.apache.hadoop.hbase.master.region.MasterRegion.open(MasterRegion.java:269)
> at
> org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:309)
> at
> org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:949)
> at
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2240)
> at
> org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:622)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25116) RegionMonitor support RegionTask count normalize
[ https://issues.apache.org/jira/browse/HBASE-25116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17225806#comment-17225806 ] niuyulin commented on HBASE-25116: -- yes, If the region num is too large , the canary slows the cluster. > RegionMonitor support RegionTask count normalize > > > Key: HBASE-25116 > URL: https://issues.apache.org/jira/browse/HBASE-25116 > Project: HBase > Issue Type: Improvement >Reporter: niuyulin >Assignee: niuyulin >Priority: Minor > > large count of region task from canary may affect user normal request, > meanwhile if region task is few, the availability monitoring may shake for > occasional exception. > so , if the task count is large , we will randomly trim tasks for each table, > according to the raito of the table region count in whole tasks region count. > If the task count is few, we will repeat tasks -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25213) Should request Compaction after bulkLoadHFiles is done
niuyulin created HBASE-25213: Summary: Should request Compaction after bulkLoadHFiles is done Key: HBASE-25213 URL: https://issues.apache.org/jira/browse/HBASE-25213 Project: HBase Issue Type: Improvement Reporter: niuyulin Assignee: niuyulin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-25210) RegionInfo.isOffline is now a duplication with RegionInfo.isSplit
[ https://issues.apache.org/jira/browse/HBASE-25210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin reassigned HBASE-25210: Assignee: niuyulin > RegionInfo.isOffline is now a duplication with RegionInfo.isSplit > - > > Key: HBASE-25210 > URL: https://issues.apache.org/jira/browse/HBASE-25210 > Project: HBase > Issue Type: Improvement >Reporter: Duo Zhang >Assignee: niuyulin >Priority: Major > > The only place, where we set it to true is in splitRegion, and at the same > time we will set split to true. > So in general, I suggest that we deprecated isOffline and isSplitParent in > RegionInfo, only leave the isSplit method. And in RegionInfoBuilder, we > deprecated setOffline and only leave the setSplit method. > This could make our code base cleaner. > And for serialization compatibility, we'd better still keep the split and > offline fields in the actual RegionInfo datastructure for a while. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25093) the RSGroupBasedLoadBalancer#retainAssignment throws NPE
[ https://issues.apache.org/jira/browse/HBASE-25093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin resolved HBASE-25093. -- Resolution: Fixed > the RSGroupBasedLoadBalancer#retainAssignment throws NPE > > > Key: HBASE-25093 > URL: https://issues.apache.org/jira/browse/HBASE-25093 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 3.0.0-alpha-1, 2.3.1, 2.2.3 >Reporter: Bo Cui >Assignee: niuyulin >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.3, 2.4.0, 2.2.7 > > > when BaseLoadBalancer# > https://github.com/apache/hbase/blob/8bfa2cb2eedcf050b26a28961e1b77dbf3cd8c95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java#L1433 > If the result of the BaseLoadBalancer#retainAssignment is null, the > RSGroupBasedLoadBalancer#retainAssignment will throw NPE. > https://github.com/apache/hbase/blob/8bfa2cb2eedcf050b26a28961e1b77dbf3cd8c95/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java#L206 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25093) the RSGroupBasedLoadBalancer#retainAssignment throws NPE
[ https://issues.apache.org/jira/browse/HBASE-25093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17216415#comment-17216415 ] niuyulin commented on HBASE-25093: -- Thanks for [~zhangduo] reviewing > the RSGroupBasedLoadBalancer#retainAssignment throws NPE > > > Key: HBASE-25093 > URL: https://issues.apache.org/jira/browse/HBASE-25093 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 3.0.0-alpha-1, 2.3.1, 2.2.3 >Reporter: Bo Cui >Assignee: niuyulin >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.3, 2.4.0, 2.2.7 > > > when BaseLoadBalancer# > https://github.com/apache/hbase/blob/8bfa2cb2eedcf050b26a28961e1b77dbf3cd8c95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java#L1433 > If the result of the BaseLoadBalancer#retainAssignment is null, the > RSGroupBasedLoadBalancer#retainAssignment will throw NPE. > https://github.com/apache/hbase/blob/8bfa2cb2eedcf050b26a28961e1b77dbf3cd8c95/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java#L206 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25201) YouAreDeadException should be moved to hbase-server module
[ https://issues.apache.org/jira/browse/HBASE-25201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin updated HBASE-25201: - Description: Since it will be thrown from the region server reports, it will not be propagated to client, we should move this class to hbase-server. > YouAreDeadException should be moved to hbase-server module > -- > > Key: HBASE-25201 > URL: https://issues.apache.org/jira/browse/HBASE-25201 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: niuyulin >Assignee: niuyulin >Priority: Major > > Since it will be thrown from the region server reports, it will not be > propagated to client, we should move this class to hbase-server. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25201) YouAreDeadException should be moved to hbase-server module
niuyulin created HBASE-25201: Summary: YouAreDeadException should be moved to hbase-server module Key: HBASE-25201 URL: https://issues.apache.org/jira/browse/HBASE-25201 Project: HBase Issue Type: Improvement Components: Client Reporter: niuyulin Assignee: niuyulin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25093) the RSGroupBasedLoadBalancer#retainAssignment throws NPE
[ https://issues.apache.org/jira/browse/HBASE-25093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17216408#comment-17216408 ] niuyulin commented on HBASE-25093: -- [~zhangduo] PR for 2.3 has submitted, mind help to review it ? > the RSGroupBasedLoadBalancer#retainAssignment throws NPE > > > Key: HBASE-25093 > URL: https://issues.apache.org/jira/browse/HBASE-25093 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 3.0.0-alpha-1, 2.3.1, 2.2.3 >Reporter: Bo Cui >Assignee: niuyulin >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.3, 2.4.0, 2.2.7 > > > when BaseLoadBalancer# > https://github.com/apache/hbase/blob/8bfa2cb2eedcf050b26a28961e1b77dbf3cd8c95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java#L1433 > If the result of the BaseLoadBalancer#retainAssignment is null, the > RSGroupBasedLoadBalancer#retainAssignment will throw NPE. > https://github.com/apache/hbase/blob/8bfa2cb2eedcf050b26a28961e1b77dbf3cd8c95/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java#L206 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25093) the RSGroupBasedLoadBalancer#retainAssignment throws NPE
[ https://issues.apache.org/jira/browse/HBASE-25093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin resolved HBASE-25093. -- Resolution: Fixed > the RSGroupBasedLoadBalancer#retainAssignment throws NPE > > > Key: HBASE-25093 > URL: https://issues.apache.org/jira/browse/HBASE-25093 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 3.0.0-alpha-1, 2.3.1, 2.2.3 >Reporter: Bo Cui >Assignee: niuyulin >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.3, 2.4.0, 2.2.7 > > > when BaseLoadBalancer# > https://github.com/apache/hbase/blob/8bfa2cb2eedcf050b26a28961e1b77dbf3cd8c95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java#L1433 > If the result of the BaseLoadBalancer#retainAssignment is null, the > RSGroupBasedLoadBalancer#retainAssignment will throw NPE. > https://github.com/apache/hbase/blob/8bfa2cb2eedcf050b26a28961e1b77dbf3cd8c95/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java#L206 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25093) the RSGroupBasedLoadBalancer#retainAssignment throws NPE
[ https://issues.apache.org/jira/browse/HBASE-25093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212791#comment-17212791 ] niuyulin commented on HBASE-25093: -- [~zhangduo] of course, I have submitted another PR for branch-2 > the RSGroupBasedLoadBalancer#retainAssignment throws NPE > > > Key: HBASE-25093 > URL: https://issues.apache.org/jira/browse/HBASE-25093 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 3.0.0-alpha-1, 2.3.1, 2.2.3 >Reporter: Bo Cui >Assignee: niuyulin >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.3, 2.4.0, 2.2.7 > > > when BaseLoadBalancer# > https://github.com/apache/hbase/blob/8bfa2cb2eedcf050b26a28961e1b77dbf3cd8c95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java#L1433 > If the result of the BaseLoadBalancer#retainAssignment is null, the > RSGroupBasedLoadBalancer#retainAssignment will throw NPE. > https://github.com/apache/hbase/blob/8bfa2cb2eedcf050b26a28961e1b77dbf3cd8c95/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java#L206 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-25176) MasterStoppedException should be moved to hbase-client module
[ https://issues.apache.org/jira/browse/HBASE-25176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin reassigned HBASE-25176: Assignee: niuyulin > MasterStoppedException should be moved to hbase-client module > - > > Key: HBASE-25176 > URL: https://issues.apache.org/jira/browse/HBASE-25176 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Duo Zhang >Assignee: niuyulin >Priority: Major > > Since it will be thrown from the HMaster.checkInitialized which will be > called in a rpc call method, which means it will be propagated to client, we > should move this class to hbase-client. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-25175) Remove the constructors of HBaseConfiguration
[ https://issues.apache.org/jira/browse/HBASE-25175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin reassigned HBASE-25175: Assignee: niuyulin > Remove the constructors of HBaseConfiguration > - > > Key: HBASE-25175 > URL: https://issues.apache.org/jira/browse/HBASE-25175 > Project: HBase > Issue Type: Sub-task > Components: API >Reporter: Duo Zhang >Assignee: niuyulin >Priority: Major > > As they have been marked as deprecated long ago. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25115) HFilePrettyPrinter can't seek to the row which is the first row of a hfile
[
https://issues.apache.org/jira/browse/HBASE-25115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17204669#comment-17204669
]
niuyulin commented on HBASE-25115:
--
[~zcq_rambo] Yes, your code is simplified, I will modify my PR on master branch
> HFilePrettyPrinter can't seek to the row which is the first row of a hfile
> --
>
> Key: HBASE-25115
> URL: https://issues.apache.org/jira/browse/HBASE-25115
> Project: HBase
> Issue Type: Bug
> Components: HFile, tooling
>Affects Versions: 2.2.6
>Reporter: Chaoqiang Zhong
>Assignee: niuyulin
>Priority: Major
> Attachments: HBASE-25115.patch
>
>
> This issue can be reproduced by below steps:
> * make a hfile contains two rows '000' and '001';
> {code:java}
> D:\bin>hbase hfile -p -f
> /hbase/data/default/test2/df76e4acab5398e70be332f6807ec3ba/f1/fda213c556d540a58d29d6bd85931dcd
> K: 000/f1:a/1601282789548/Put/vlen=4/seqid=4 V:
> K: 001/f1:a/1601282792779/Put/vlen=4/seqid=5 V:
> Scanned kv count -> 2{code}
> * '001' can be seeked to;
> {code:java}
> D:\bin>hbase hfile -e -w 001 -f
> /hbase/data/default/test2/df76e4acab5398e70be332f6807ec3ba/f1/fda213c556d540a58d29d6bd85931dcd
> K: 001/f1:a/1601282792779/Put/vlen=4/seqid=5
> Scanned kv count -> 1{code}
> * but '000' can't be seeked to;
> {code:java}
> D:\bin>hbase hfile -e -w 000 -f
> /hbase/data/default/test2/df76e4acab5398e70be332f6807ec3ba/f1/fda213c556d540a58d29d6bd85931dcd
> Scanned kv count -> 0{code}
> In HFilePrettyPrinter we use
> "scanner.seekTo(PrivateCellUtil.createFirstOnRow(this.row))" to seek to
> row.But this method will retrurn -1 when the row is the first row of hfile.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25116) RegionMonitor support RegionTask count normalize
[ https://issues.apache.org/jira/browse/HBASE-25116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin updated HBASE-25116: - Description: large count of region task from canary may affect user normal request, meanwhile if region task is few, the availability monitoring may shake for occasional exception. so , if the task count is large , we will randomly trim tasks for each table, according to the raito of the table region count in whole tasks region count. If the task count is few, we will repeat tasks was: large count of region task from canary may affect user normal request, meanwhile if region task is few, the availability monitoring may shake for occasional exception. so , if the task count is large , we will randomly trim tasks for each table, according to the raito of the table region count in whole tasks region count. if the task count is few, we will repeat tasks > RegionMonitor support RegionTask count normalize > > > Key: HBASE-25116 > URL: https://issues.apache.org/jira/browse/HBASE-25116 > Project: HBase > Issue Type: Improvement >Reporter: niuyulin >Assignee: niuyulin >Priority: Minor > > large count of region task from canary may affect user normal request, > meanwhile if region task is few, the availability monitoring may shake for > occasional exception. > so , if the task count is large , we will randomly trim tasks for each table, > according to the raito of the table region count in whole tasks region count. > If the task count is few, we will repeat tasks -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25116) RegionMonitor support RegionTask count normalize
[ https://issues.apache.org/jira/browse/HBASE-25116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin updated HBASE-25116: - Description: large count of region task from canary may affect user normal request, meanwhile if region task is few, the availability monitoring may shake for occasional exception. so , if the task count is large , we will randomly trim tasks for each table, according to the raito of the table region count in whole tasks region count. if the task count is few, we will repeat tasks was:large count of region task from canary may affect user normal request, meanwhile if region task is few, the availability monitoring may shake for occasional exception > RegionMonitor support RegionTask count normalize > > > Key: HBASE-25116 > URL: https://issues.apache.org/jira/browse/HBASE-25116 > Project: HBase > Issue Type: Improvement >Reporter: niuyulin >Assignee: niuyulin >Priority: Minor > > large count of region task from canary may affect user normal request, > meanwhile if region task is few, the availability monitoring may shake for > occasional exception. > so , if the task count is large , we will randomly trim tasks for each table, > according to the raito of the table region count in whole tasks region count. > if the task count is few, we will repeat tasks -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25116) RegionMonitor support RegionTask count normalize
[ https://issues.apache.org/jira/browse/HBASE-25116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin updated HBASE-25116: - Description: large count of region task from canary may affect user normal request, meanwhile if region task is few, the availability monitoring may shake for occasional exception > RegionMonitor support RegionTask count normalize > > > Key: HBASE-25116 > URL: https://issues.apache.org/jira/browse/HBASE-25116 > Project: HBase > Issue Type: Improvement >Reporter: niuyulin >Assignee: niuyulin >Priority: Minor > > large count of region task from canary may affect user normal request, > meanwhile if region task is few, the availability monitoring may shake for > occasional exception -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25116) RegionMonitor support RegionTask count normalize
niuyulin created HBASE-25116: Summary: RegionMonitor support RegionTask count normalize Key: HBASE-25116 URL: https://issues.apache.org/jira/browse/HBASE-25116 Project: HBase Issue Type: Improvement Reporter: niuyulin Assignee: niuyulin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-25115) HFilePrettyPrinter can't seek to the row which is the first row of a hfile
[
https://issues.apache.org/jira/browse/HBASE-25115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niuyulin reassigned HBASE-25115:
Assignee: niuyulin
> HFilePrettyPrinter can't seek to the row which is the first row of a hfile
> --
>
> Key: HBASE-25115
> URL: https://issues.apache.org/jira/browse/HBASE-25115
> Project: HBase
> Issue Type: Bug
> Components: HFile, tooling
>Affects Versions: 2.2.6
>Reporter: Chaoqiang Zhong
>Assignee: niuyulin
>Priority: Major
>
> This issue can be reproduced by below steps:
> * make a hfile contains two rows '000' and '001';
> {code:java}
> D:\bin>hbase hfile -p -f
> /hbase/data/default/test2/df76e4acab5398e70be332f6807ec3ba/f1/fda213c556d540a58d29d6bd85931dcd
> K: 000/f1:a/1601282789548/Put/vlen=4/seqid=4 V:
> K: 001/f1:a/1601282792779/Put/vlen=4/seqid=5 V:
> Scanned kv count -> 2{code}
> * '001' can be seeked to;
> {code:java}
> D:\bin>hbase hfile -e -w 001 -f
> /hbase/data/default/test2/df76e4acab5398e70be332f6807ec3ba/f1/fda213c556d540a58d29d6bd85931dcd
> K: 001/f1:a/1601282792779/Put/vlen=4/seqid=5
> Scanned kv count -> 1{code}
> * but '000' can't be seeked to;
> {code:java}
> D:\bin>hbase hfile -e -w 000 -f
> /hbase/data/default/test2/df76e4acab5398e70be332f6807ec3ba/f1/fda213c556d540a58d29d6bd85931dcd
> Scanned kv count -> 0{code}
> In HFilePrettyPrinter we use
> "scanner.seekTo(PrivateCellUtil.createFirstOnRow(this.row))" to seek to
> row.But this method will retrurn -1 when the row is the first row of hfile.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-25093) the RSGroupBasedLoadBalancer#retainAssignment throws NPE
[ https://issues.apache.org/jira/browse/HBASE-25093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin reassigned HBASE-25093: Assignee: niuyulin > the RSGroupBasedLoadBalancer#retainAssignment throws NPE > > > Key: HBASE-25093 > URL: https://issues.apache.org/jira/browse/HBASE-25093 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 3.0.0-alpha-1, 2.3.1, 2.2.3 >Reporter: Bo Cui >Assignee: niuyulin >Priority: Major > > when BaseLoadBalancer# > https://github.com/apache/hbase/blob/8bfa2cb2eedcf050b26a28961e1b77dbf3cd8c95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java#L1433 > If the result of the BaseLoadBalancer#retainAssignment is null, the > RSGroupBasedLoadBalancer#retainAssignment will throw NPE. > https://github.com/apache/hbase/blob/8bfa2cb2eedcf050b26a28961e1b77dbf3cd8c95/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java#L206 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25076) fix typo in MasterRegion java doc
[
https://issues.apache.org/jira/browse/HBASE-25076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niuyulin updated HBASE-25076:
-
Description: The archived HFile will be moved to the global HFile archived
directory with the \{@link MasterRegionParams#archivedWalSuffix()} suffix. The
archived WAL will be moved to the global WAL archived directory with the
\{@link MasterRegionParams#archivedHFileSuffix()} suffix (was: The archived
HFile will be moved to the global HFile archived directory with the \{@link
MasterRegionParams#archivedWalSuffix()} suffix
The archived WAL will be moved to the global WAL archived directory with the
\{@link MasterRegionParams#archivedHFileSuffix()} suffix)
> fix typo in MasterRegion java doc
> -
>
> Key: HBASE-25076
> URL: https://issues.apache.org/jira/browse/HBASE-25076
> Project: HBase
> Issue Type: Improvement
>Reporter: niuyulin
>Assignee: niuyulin
>Priority: Major
>
> The archived HFile will be moved to the global HFile archived directory with
> the \{@link MasterRegionParams#archivedWalSuffix()} suffix. The archived WAL
> will be moved to the global WAL archived directory with the \{@link
> MasterRegionParams#archivedHFileSuffix()} suffix
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25076) fix typo in MasterRegion java doc
[
https://issues.apache.org/jira/browse/HBASE-25076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niuyulin updated HBASE-25076:
-
Description:
The archived HFile will be moved to the global HFile archived directory with
the \{@link MasterRegionParams#archivedWalSuffix()} suffix
The archived WAL will be moved to the global WAL archived directory with the
\{@link MasterRegionParams#archivedHFileSuffix()} suffix
> fix typo in MasterRegion java doc
> -
>
> Key: HBASE-25076
> URL: https://issues.apache.org/jira/browse/HBASE-25076
> Project: HBase
> Issue Type: Improvement
>Reporter: niuyulin
>Assignee: niuyulin
>Priority: Major
>
> The archived HFile will be moved to the global HFile archived directory with
> the \{@link MasterRegionParams#archivedWalSuffix()} suffix
> The archived WAL will be moved to the global WAL archived directory with the
> \{@link MasterRegionParams#archivedHFileSuffix()} suffix
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25076) fix typo in MasterRegion java doc
niuyulin created HBASE-25076: Summary: fix typo in MasterRegion java doc Key: HBASE-25076 URL: https://issues.apache.org/jira/browse/HBASE-25076 Project: HBase Issue Type: Improvement Reporter: niuyulin Assignee: niuyulin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-25075) Fix typo in ReplicationProtobufUtil
[ https://issues.apache.org/jira/browse/HBASE-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin reassigned HBASE-25075: Assignee: niuyulin > Fix typo in ReplicationProtobufUtil > --- > > Key: HBASE-25075 > URL: https://issues.apache.org/jira/browse/HBASE-25075 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Duo Zhang >Assignee: niuyulin >Priority: Major > Labels: beginner, trivial > > It is named as 'ReplicationProtbufUtil', missed an 'o'. > It is IA.Private so it is safe to rename it directly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-25072) Remove the unnecessary System.out.println in MasterRegistry
[ https://issues.apache.org/jira/browse/HBASE-25072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin reassigned HBASE-25072: Assignee: niuyulin > Remove the unnecessary System.out.println in MasterRegistry > --- > > Key: HBASE-25072 > URL: https://issues.apache.org/jira/browse/HBASE-25072 > Project: HBase > Issue Type: Task > Components: Client >Reporter: Duo Zhang >Assignee: niuyulin >Priority: Minor > Labels: beginner, trivial > > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/MasterRegistry.java#L328 > This line. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-25053) WAL replay should ignore 0-length files
[
https://issues.apache.org/jira/browse/HBASE-25053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niuyulin reassigned HBASE-25053:
Assignee: niuyulin
> WAL replay should ignore 0-length files
> ---
>
> Key: HBASE-25053
> URL: https://issues.apache.org/jira/browse/HBASE-25053
> Project: HBase
> Issue Type: Bug
> Components: master, regionserver
>Affects Versions: 2.3.1
>Reporter: Nick Dimiduk
>Assignee: niuyulin
>Priority: Major
>
> I overdrove a small testing cluster, filling HDFS. After cleaning up data to
> bring HBase back up, I noticed all masters -refused to start- abort. Logs
> complain of seeking past EOF. Indeed the last wal file name logged is a
> 0-length file. WAL replay should gracefully skip and clean up such an empty
> file.
> {noformat}
> 2020-09-16 19:51:30,297 ERROR org.apache.hadoop.hbase.master.HMaster: Failed
> to become active master
> java.io.EOFException: Cannot seek after EOF
> at
> org.apache.hadoop.hdfs.DFSInputStream.seek(DFSInputStream.java:1448)
> at
> org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:66)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initInternal(ProtobufLogReader.java:211)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initReader(ProtobufLogReader.java:173)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:64)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:168)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:323)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:305)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:293)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:429)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:4859)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:4765)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:1014)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:956)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7496)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegionFromTableDir(HRegion.java:7454)
> at
> org.apache.hadoop.hbase.master.region.MasterRegion.open(MasterRegion.java:269)
> at
> org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:309)
> at
> org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:949)
> at
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2240)
> at
> org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:622)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25016) Should close ResultScanner in MetaTableAccessor.scanByRegionEncodedName
[ https://issues.apache.org/jira/browse/HBASE-25016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195164#comment-17195164 ] niuyulin commented on HBASE-25016: -- tanks [~zhangduo] and [~vjasani] , I will submit an addendum PR. > Should close ResultScanner in MetaTableAccessor.scanByRegionEncodedName > --- > > Key: HBASE-25016 > URL: https://issues.apache.org/jira/browse/HBASE-25016 > Project: HBase > Issue Type: Bug > Components: master, meta >Reporter: Duo Zhang >Assignee: niuyulin >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.4.0, 2.3.2 > > > It will cause resource leak. > Th good news is that this method is only used for cluster maintainance, such > as assign or unassign region manually, so it is less hurt. Anyway, we should > fix it ASAP. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-25016) Should close ResultScanner in MetaTableAccessor.scanByRegionEncodedName
[ https://issues.apache.org/jira/browse/HBASE-25016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195164#comment-17195164 ] niuyulin edited comment on HBASE-25016 at 9/14/20, 2:56 AM: Thanks [~zhangduo] and [~vjasani] , I will submit an addendum PR. was (Author: niuyulin): tanks [~zhangduo] and [~vjasani] , I will submit an addendum PR. > Should close ResultScanner in MetaTableAccessor.scanByRegionEncodedName > --- > > Key: HBASE-25016 > URL: https://issues.apache.org/jira/browse/HBASE-25016 > Project: HBase > Issue Type: Bug > Components: master, meta >Reporter: Duo Zhang >Assignee: niuyulin >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.4.0, 2.3.2 > > > It will cause resource leak. > Th good news is that this method is only used for cluster maintainance, such > as assign or unassign region manually, so it is less hurt. Anyway, we should > fix it ASAP. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-25016) Should close ResultScanner in MetaTableAccessor.scanByRegionEncodedName
[ https://issues.apache.org/jira/browse/HBASE-25016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin reassigned HBASE-25016: Assignee: niuyulin > Should close ResultScanner in MetaTableAccessor.scanByRegionEncodedName > --- > > Key: HBASE-25016 > URL: https://issues.apache.org/jira/browse/HBASE-25016 > Project: HBase > Issue Type: Bug > Components: master, meta >Reporter: Duo Zhang >Assignee: niuyulin >Priority: Critical > > It will cause resource leak. > Th good news is that this method is only used for cluster maintainance, such > as assign or unassign region manually, so it is less hurt. Anyway, we should > fix it ASAP. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24993) Remove OfflineMetaRebuildTestCore
[ https://issues.apache.org/jira/browse/HBASE-24993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin resolved HBASE-24993. -- Fix Version/s: 3.0.0-alpha-1 Resolution: Fixed > Remove OfflineMetaRebuildTestCore > - > > Key: HBASE-24993 > URL: https://issues.apache.org/jira/browse/HBASE-24993 > Project: HBase > Issue Type: Task > Components: test >Reporter: Duo Zhang >Assignee: niuyulin >Priority: Major > Fix For: 3.0.0-alpha-1 > > > Maybe it is used to test HBCK but is useless now. Let's remove it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24993) Remove OfflineMetaRebuildTestCore
[ https://issues.apache.org/jira/browse/HBASE-24993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191751#comment-17191751 ] niuyulin edited comment on HBASE-24993 at 9/8/20, 2:28 AM: --- Yes, We have FsRegionsMetaRecoverer/TestFsRegionsMetaRecoverer, let me remove this useless class. was (Author: niuyulin): let me remove it > Remove OfflineMetaRebuildTestCore > - > > Key: HBASE-24993 > URL: https://issues.apache.org/jira/browse/HBASE-24993 > Project: HBase > Issue Type: Task > Components: test >Reporter: Duo Zhang >Assignee: niuyulin >Priority: Major > > Maybe it is used to test HBCK but is useless now. Let's remove it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-24993) Remove OfflineMetaRebuildTestCore
[ https://issues.apache.org/jira/browse/HBASE-24993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin reassigned HBASE-24993: Assignee: niuyulin > Remove OfflineMetaRebuildTestCore > - > > Key: HBASE-24993 > URL: https://issues.apache.org/jira/browse/HBASE-24993 > Project: HBase > Issue Type: Task > Components: test >Reporter: Duo Zhang >Assignee: niuyulin >Priority: Major > > Maybe it is used to test HBCK but is useless now. Let's remove it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24993) Remove OfflineMetaRebuildTestCore
[ https://issues.apache.org/jira/browse/HBASE-24993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191751#comment-17191751 ] niuyulin commented on HBASE-24993: -- let me remove it > Remove OfflineMetaRebuildTestCore > - > > Key: HBASE-24993 > URL: https://issues.apache.org/jira/browse/HBASE-24993 > Project: HBase > Issue Type: Task > Components: test >Reporter: Duo Zhang >Priority: Major > > Maybe it is used to test HBCK but is useless now. Let's remove it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24928) balanceRSGroup should skip generating balance plan for disabled table and splitParent region
[
https://issues.apache.org/jira/browse/HBASE-24928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183128#comment-17183128
]
niuyulin commented on HBASE-24928:
--
[~zghao] Yes, I have submitted another PR
> balanceRSGroup should skip generating balance plan for disabled table and
> splitParent region
>
>
> Key: HBASE-24928
> URL: https://issues.apache.org/jira/browse/HBASE-24928
> Project: HBase
> Issue Type: Improvement
> Components: Balancer
>Reporter: niuyulin
>Assignee: niuyulin
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> now ,we generate balance plan for disabled tables, which is useless
> {code:java}
> 2020-08-20,20:47:54,702 WARN
> [RpcServer.default.RWQ.Fifo.read.handler=310,queue=6,port=22500]
> org.apache.hadoop.hbase.master.HMaster: Failed balance plan:
> hri=aa325467924edc865ab2ef6d82f9e2a7,
> source=tj1-hadoop-staging-st02.kscn,22600,1572403947348, destination=, just
> skip it
> org.apache.hadoop.hbase.client.DoNotRetryRegionException: Unexpected state
> for rit=CLOSED, location=tj1-hadoop-staging-st02.kscn,22600,1572403947348,
> table=galaxysds:sds_staging_258z, region=aa325467924edc865ab2ef6d82f9e2a7
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.preTransitCheck(AssignmentManager.java:580)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createMoveRegionProcedure(AssignmentManager.java:635)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.moveAsync(AssignmentManager.java:652)
> at
> org.apache.hadoop.hbase.master.HMaster.executeRegionPlansWithThrottling(HMaster.java:1776)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.balanceRSGroup(RSGroupAdminServer.java:486)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl.balanceRSGroup(RSGroupAdminEndpoint.java:293)
> at
> org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos$RSGroupAdminService.callMethod(RSGroupAdminProtos.java:13890)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:908)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:135)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24928) balanceRSGroup should skip generating balance plan for disabled table and splitParent region
[
https://issues.apache.org/jira/browse/HBASE-24928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niuyulin updated HBASE-24928:
-
Summary: balanceRSGroup should skip generating balance plan for disabled
table and splitParent region (was: balanceRSGroup should skip generating
balance plan for disabled table)
> balanceRSGroup should skip generating balance plan for disabled table and
> splitParent region
>
>
> Key: HBASE-24928
> URL: https://issues.apache.org/jira/browse/HBASE-24928
> Project: HBase
> Issue Type: Improvement
> Components: Balancer
>Reporter: niuyulin
>Assignee: niuyulin
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> now ,we generate balance plan for disabled tables, which is useless
> {code:java}
> 2020-08-20,20:47:54,702 WARN
> [RpcServer.default.RWQ.Fifo.read.handler=310,queue=6,port=22500]
> org.apache.hadoop.hbase.master.HMaster: Failed balance plan:
> hri=aa325467924edc865ab2ef6d82f9e2a7,
> source=tj1-hadoop-staging-st02.kscn,22600,1572403947348, destination=, just
> skip it
> org.apache.hadoop.hbase.client.DoNotRetryRegionException: Unexpected state
> for rit=CLOSED, location=tj1-hadoop-staging-st02.kscn,22600,1572403947348,
> table=galaxysds:sds_staging_258z, region=aa325467924edc865ab2ef6d82f9e2a7
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.preTransitCheck(AssignmentManager.java:580)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createMoveRegionProcedure(AssignmentManager.java:635)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.moveAsync(AssignmentManager.java:652)
> at
> org.apache.hadoop.hbase.master.HMaster.executeRegionPlansWithThrottling(HMaster.java:1776)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.balanceRSGroup(RSGroupAdminServer.java:486)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl.balanceRSGroup(RSGroupAdminEndpoint.java:293)
> at
> org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos$RSGroupAdminService.callMethod(RSGroupAdminProtos.java:13890)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:908)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:135)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24928) balanceRSGroup should skip generating balance plan for disabled table
[
https://issues.apache.org/jira/browse/HBASE-24928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niuyulin updated HBASE-24928:
-
Description:
now ,we generate balance plan for disabled tables, which is useless
{code:java}
2020-08-20,20:47:54,702 WARN
[RpcServer.default.RWQ.Fifo.read.handler=310,queue=6,port=22500]
org.apache.hadoop.hbase.master.HMaster: Failed balance plan:
hri=aa325467924edc865ab2ef6d82f9e2a7,
source=tj1-hadoop-staging-st02.kscn,22600,1572403947348, destination=, just
skip it
org.apache.hadoop.hbase.client.DoNotRetryRegionException: Unexpected state for
rit=CLOSED, location=tj1-hadoop-staging-st02.kscn,22600,1572403947348,
table=galaxysds:sds_staging_258z, region=aa325467924edc865ab2ef6d82f9e2a7
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.preTransitCheck(AssignmentManager.java:580)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.createMoveRegionProcedure(AssignmentManager.java:635)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.moveAsync(AssignmentManager.java:652)
at
org.apache.hadoop.hbase.master.HMaster.executeRegionPlansWithThrottling(HMaster.java:1776)
at
org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.balanceRSGroup(RSGroupAdminServer.java:486)
at
org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl.balanceRSGroup(RSGroupAdminEndpoint.java:293)
at
org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos$RSGroupAdminService.callMethod(RSGroupAdminProtos.java:13890)
at
org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:908)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:135)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
{code}
was:
{code:java}
2020-08-20,20:47:54,702 WARN
[RpcServer.default.RWQ.Fifo.read.handler=310,queue=6,port=22500]
org.apache.hadoop.hbase.master.HMaster: Failed balance plan:
hri=aa325467924edc865ab2ef6d82f9e2a7,
source=tj1-hadoop-staging-st02.kscn,22600,1572403947348, destination=, just
skip it
org.apache.hadoop.hbase.client.DoNotRetryRegionException: Unexpected state for
rit=CLOSED, location=tj1-hadoop-staging-st02.kscn,22600,1572403947348,
table=galaxysds:sds_staging_258z, region=aa325467924edc865ab2ef6d82f9e2a7
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.preTransitCheck(AssignmentManager.java:580)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.createMoveRegionProcedure(AssignmentManager.java:635)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.moveAsync(AssignmentManager.java:652)
at
org.apache.hadoop.hbase.master.HMaster.executeRegionPlansWithThrottling(HMaster.java:1776)
at
org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.balanceRSGroup(RSGroupAdminServer.java:486)
at
org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl.balanceRSGroup(RSGroupAdminEndpoint.java:293)
at
org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos$RSGroupAdminService.callMethod(RSGroupAdminProtos.java:13890)
at
org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:908)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:135)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
{code}
> balanceRSGroup should skip generating balance plan for disabled table
> -
>
> Key: HBASE-24928
> URL: https://issues.apache.org/jira/browse/HBASE-24928
> Project: HBase
> Issue Type: Improvement
> Components: Balancer
>Reporter: niuyulin
>Assignee: niuyulin
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> now ,we generate balance plan for disabled tables, which is useless
> {code:java}
> 2020-08-20,20:47:54,702 WARN
> [RpcServer.default.RWQ.Fifo.read.handler=310,queue=6,port=22500]
> org.apache.hadoop.hbase.master.HMaster: Failed balance plan:
> hri=aa325467924edc865ab2ef6d82f9e2a7,
> source=tj1-hadoop-staging-st02.kscn,22600,1572403947348, destination=, just
> skip it
>
[jira] [Updated] (HBASE-24928) balanceRSGroup should skip generating balance plan for disabled table
[
https://issues.apache.org/jira/browse/HBASE-24928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niuyulin updated HBASE-24928:
-
Description:
{code:java}
2020-08-20,20:47:54,702 WARN
[RpcServer.default.RWQ.Fifo.read.handler=310,queue=6,port=22500]
org.apache.hadoop.hbase.master.HMaster: Failed balance plan:
hri=aa325467924edc865ab2ef6d82f9e2a7,
source=tj1-hadoop-staging-st02.kscn,22600,1572403947348, destination=, just
skip it
org.apache.hadoop.hbase.client.DoNotRetryRegionException: Unexpected state for
rit=CLOSED, location=tj1-hadoop-staging-st02.kscn,22600,1572403947348,
table=galaxysds:sds_staging_258z, region=aa325467924edc865ab2ef6d82f9e2a7
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.preTransitCheck(AssignmentManager.java:580)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.createMoveRegionProcedure(AssignmentManager.java:635)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.moveAsync(AssignmentManager.java:652)
at
org.apache.hadoop.hbase.master.HMaster.executeRegionPlansWithThrottling(HMaster.java:1776)
at
org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.balanceRSGroup(RSGroupAdminServer.java:486)
at
org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl.balanceRSGroup(RSGroupAdminEndpoint.java:293)
at
org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos$RSGroupAdminService.callMethod(RSGroupAdminProtos.java:13890)
at
org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:908)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:135)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
{code}
> balanceRSGroup should skip generating balance plan for disabled table
> -
>
> Key: HBASE-24928
> URL: https://issues.apache.org/jira/browse/HBASE-24928
> Project: HBase
> Issue Type: Improvement
> Components: Balancer
>Reporter: niuyulin
>Assignee: niuyulin
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> {code:java}
> 2020-08-20,20:47:54,702 WARN
> [RpcServer.default.RWQ.Fifo.read.handler=310,queue=6,port=22500]
> org.apache.hadoop.hbase.master.HMaster: Failed balance plan:
> hri=aa325467924edc865ab2ef6d82f9e2a7,
> source=tj1-hadoop-staging-st02.kscn,22600,1572403947348, destination=, just
> skip it
> org.apache.hadoop.hbase.client.DoNotRetryRegionException: Unexpected state
> for rit=CLOSED, location=tj1-hadoop-staging-st02.kscn,22600,1572403947348,
> table=galaxysds:sds_staging_258z, region=aa325467924edc865ab2ef6d82f9e2a7
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.preTransitCheck(AssignmentManager.java:580)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createMoveRegionProcedure(AssignmentManager.java:635)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.moveAsync(AssignmentManager.java:652)
> at
> org.apache.hadoop.hbase.master.HMaster.executeRegionPlansWithThrottling(HMaster.java:1776)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.balanceRSGroup(RSGroupAdminServer.java:486)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl.balanceRSGroup(RSGroupAdminEndpoint.java:293)
> at
> org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos$RSGroupAdminService.callMethod(RSGroupAdminProtos.java:13890)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:908)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:135)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24928) balanceRSGroup should skip generating balance plan for disabled table
[ https://issues.apache.org/jira/browse/HBASE-24928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin updated HBASE-24928: - Component/s: Balancer Fix Version/s: 3.0.0-alpha-1 > balanceRSGroup should skip generating balance plan for disabled table > - > > Key: HBASE-24928 > URL: https://issues.apache.org/jira/browse/HBASE-24928 > Project: HBase > Issue Type: Improvement > Components: Balancer >Reporter: niuyulin >Assignee: niuyulin >Priority: Major > Fix For: 3.0.0-alpha-1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24928) balanceRSGroup should skip generating balance plan for disabled table
niuyulin created HBASE-24928: Summary: balanceRSGroup should skip generating balance plan for disabled table Key: HBASE-24928 URL: https://issues.apache.org/jira/browse/HBASE-24928 Project: HBase Issue Type: Improvement Reporter: niuyulin Assignee: niuyulin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24872) refactor valueOf PoolType
[ https://issues.apache.org/jira/browse/HBASE-24872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin resolved HBASE-24872. -- Resolution: Fixed > refactor valueOf PoolType > - > > Key: HBASE-24872 > URL: https://issues.apache.org/jira/browse/HBASE-24872 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: niuyulin >Assignee: niuyulin >Priority: Minor > Fix For: 3.0.0-alpha-1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24872) refactor valueOf PoolType
[ https://issues.apache.org/jira/browse/HBASE-24872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin updated HBASE-24872: - Component/s: Client Fix Version/s: 3.0.0-alpha-1 > refactor valueOf PoolType > - > > Key: HBASE-24872 > URL: https://issues.apache.org/jira/browse/HBASE-24872 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: niuyulin >Assignee: niuyulin >Priority: Minor > Fix For: 3.0.0-alpha-1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-24886) Remove deprecated methods in RowMutations
[ https://issues.apache.org/jira/browse/HBASE-24886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin reassigned HBASE-24886: Assignee: niuyulin > Remove deprecated methods in RowMutations > - > > Key: HBASE-24886 > URL: https://issues.apache.org/jira/browse/HBASE-24886 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Duo Zhang >Assignee: niuyulin >Priority: Major > > Such as add(Put) and add(Delete). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-24880) Remove ReplicationPeerConfigUpgrader
[ https://issues.apache.org/jira/browse/HBASE-24880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin reassigned HBASE-24880: Assignee: niuyulin > Remove ReplicationPeerConfigUpgrader > > > Key: HBASE-24880 > URL: https://issues.apache.org/jira/browse/HBASE-24880 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Duo Zhang >Assignee: niuyulin >Priority: Major > > The comment says it will be removed in 3.x. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24880) Remove ReplicationPeerConfigUpgrader
[ https://issues.apache.org/jira/browse/HBASE-24880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17177482#comment-17177482 ] niuyulin commented on HBASE-24880: -- let me remove it > Remove ReplicationPeerConfigUpgrader > > > Key: HBASE-24880 > URL: https://issues.apache.org/jira/browse/HBASE-24880 > Project: HBase > Issue Type: Task > Components: Replication >Reporter: Duo Zhang >Priority: Major > > The comment says it will be removed in 3.x. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-24872) refactor valueOf PoolType
[ https://issues.apache.org/jira/browse/HBASE-24872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin reassigned HBASE-24872: Assignee: niuyulin > refactor valueOf PoolType > - > > Key: HBASE-24872 > URL: https://issues.apache.org/jira/browse/HBASE-24872 > Project: HBase > Issue Type: Improvement >Reporter: niuyulin >Assignee: niuyulin >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24872) refactor valueOf PoolType
[ https://issues.apache.org/jira/browse/HBASE-24872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin updated HBASE-24872: - Summary: refactor valueOf PoolType (was: re) > refactor valueOf PoolType > - > > Key: HBASE-24872 > URL: https://issues.apache.org/jira/browse/HBASE-24872 > Project: HBase > Issue Type: Improvement >Reporter: niuyulin >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24872) re
niuyulin created HBASE-24872: Summary: re Key: HBASE-24872 URL: https://issues.apache.org/jira/browse/HBASE-24872 Project: HBase Issue Type: Improvement Reporter: niuyulin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24825) Add UncaughtExceptionHandler for NettyRpcConnection Relogin thread
[ https://issues.apache.org/jira/browse/HBASE-24825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin resolved HBASE-24825. -- Resolution: Won't Fix [#2231|https://github.com/apache/hbase/pull/2231] has resolved this issue > Add UncaughtExceptionHandler for NettyRpcConnection Relogin thread > -- > > Key: HBASE-24825 > URL: https://issues.apache.org/jira/browse/HBASE-24825 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: niuyulin >Assignee: niuyulin >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24825) Add UncaughtExceptionHandler for NettyRpcConnection Relogin thread
[ https://issues.apache.org/jira/browse/HBASE-24825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin updated HBASE-24825: - Summary: Add UncaughtExceptionHandler for NettyRpcConnection Relogin thread (was: NettyRpcConnection.scheduleRelogin method should catch Throwable exception) > Add UncaughtExceptionHandler for NettyRpcConnection Relogin thread > -- > > Key: HBASE-24825 > URL: https://issues.apache.org/jira/browse/HBASE-24825 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: niuyulin >Assignee: niuyulin >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24659) Calculate FIXED_OVERHEAD automatically
[ https://issues.apache.org/jira/browse/HBASE-24659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17173120#comment-17173120 ] niuyulin commented on HBASE-24659: -- [~zhangduo] [~stack] Thanks for your review > Calculate FIXED_OVERHEAD automatically > -- > > Key: HBASE-24659 > URL: https://issues.apache.org/jira/browse/HBASE-24659 > Project: HBase > Issue Type: Improvement >Reporter: Duo Zhang >Assignee: niuyulin >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > Attachments: 50M-performance.pdf > > > Now the FIXED_OVERHEAD in some classes are maintained manually, an we have a > method to TestHeapSizes to confirm that the value is correct. > But it is really hard for developers to count the fields in a complicated > class like HRegion. Since we have the ability to calcuate the accurate size > in UT, I think we it is also possible to calcuate it when loading the class, > which is a one time operation so should not effect the performance too much. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24827) BackPort HBASE-11554 Remove Reusable poolmap Rpc client type.
niuyulin created HBASE-24827: Summary: BackPort HBASE-11554 Remove Reusable poolmap Rpc client type. Key: HBASE-24827 URL: https://issues.apache.org/jira/browse/HBASE-24827 Project: HBase Issue Type: Improvement Components: Client Reporter: niuyulin Assignee: niuyulin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24825) NettyRpcConnection.scheduleRelogin method should catch Throwable exception
niuyulin created HBASE-24825: Summary: NettyRpcConnection.scheduleRelogin method should catch Throwable exception Key: HBASE-24825 URL: https://issues.apache.org/jira/browse/HBASE-24825 Project: HBase Issue Type: Improvement Components: Client Reporter: niuyulin Assignee: niuyulin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24659) Calculate FIXED_OVERHEAD automatically
[ https://issues.apache.org/jira/browse/HBASE-24659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166826#comment-17166826 ] niuyulin commented on HBASE-24659: -- [~zhangduo], I do performance test via YCSB, the version of client and server are both 3.0.0-SNAPSHOT. the different between w/ and w/o this pr is mostly within 10% > Calculate FIXED_OVERHEAD automatically > -- > > Key: HBASE-24659 > URL: https://issues.apache.org/jira/browse/HBASE-24659 > Project: HBase > Issue Type: Improvement >Reporter: Duo Zhang >Assignee: niuyulin >Priority: Major > Attachments: 50M-performance.pdf > > > Now the FIXED_OVERHEAD in some classes are maintained manually, an we have a > method to TestHeapSizes to confirm that the value is correct. > But it is really hard for developers to count the fields in a complicated > class like HRegion. Since we have the ability to calcuate the accurate size > in UT, I think we it is also possible to calcuate it when loading the class, > which is a one time operation so should not effect the performance too much. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24659) Calculate FIXED_OVERHEAD automatically
[ https://issues.apache.org/jira/browse/HBASE-24659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin updated HBASE-24659: - Attachment: 50M-performance.pdf > Calculate FIXED_OVERHEAD automatically > -- > > Key: HBASE-24659 > URL: https://issues.apache.org/jira/browse/HBASE-24659 > Project: HBase > Issue Type: Improvement >Reporter: Duo Zhang >Assignee: niuyulin >Priority: Major > Attachments: 50M-performance.pdf > > > Now the FIXED_OVERHEAD in some classes are maintained manually, an we have a > method to TestHeapSizes to confirm that the value is correct. > But it is really hard for developers to count the fields in a complicated > class like HRegion. Since we have the ability to calcuate the accurate size > in UT, I think we it is also possible to calcuate it when loading the class, > which is a one time operation so should not effect the performance too much. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24653) Show snapshot owner on Master WebUI
[ https://issues.apache.org/jira/browse/HBASE-24653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17153406#comment-17153406 ] niuyulin commented on HBASE-24653: -- home page: !snapshot1.png! snapshot detail page: !snapshot2.png! > Show snapshot owner on Master WebUI > --- > > Key: HBASE-24653 > URL: https://issues.apache.org/jira/browse/HBASE-24653 > Project: HBase > Issue Type: Improvement >Reporter: Yi Mei >Assignee: niuyulin >Priority: Major > Attachments: snapshot1.png, snapshot2.png > > > Now Master UI shows lots of snapshot informations, and owner is also useful > to find out who create this snapshot. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24653) Show snapshot owner on Master WebUI
[ https://issues.apache.org/jira/browse/HBASE-24653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin updated HBASE-24653: - Attachment: snapshot2.png > Show snapshot owner on Master WebUI > --- > > Key: HBASE-24653 > URL: https://issues.apache.org/jira/browse/HBASE-24653 > Project: HBase > Issue Type: Improvement >Reporter: Yi Mei >Assignee: niuyulin >Priority: Major > Attachments: snapshot1.png, snapshot2.png > > > Now Master UI shows lots of snapshot informations, and owner is also useful > to find out who create this snapshot. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24653) Show snapshot owner on Master WebUI
[ https://issues.apache.org/jira/browse/HBASE-24653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin updated HBASE-24653: - Attachment: snapshot1.png > Show snapshot owner on Master WebUI > --- > > Key: HBASE-24653 > URL: https://issues.apache.org/jira/browse/HBASE-24653 > Project: HBase > Issue Type: Improvement >Reporter: Yi Mei >Assignee: niuyulin >Priority: Major > Attachments: snapshot1.png > > > Now Master UI shows lots of snapshot informations, and owner is also useful > to find out who create this snapshot. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24669) Logging of ppid should be consistent across all occurrences
[
https://issues.apache.org/jira/browse/HBASE-24669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17151548#comment-17151548
]
niuyulin commented on HBASE-24669:
--
hi, [~ndimiduk] , I submit a PR, could you help review it
> Logging of ppid should be consistent across all occurrences
> ---
>
> Key: HBASE-24669
> URL: https://issues.apache.org/jira/browse/HBASE-24669
> Project: HBase
> Issue Type: Improvement
> Components: Operability, proc-v2
>Reporter: Nick Dimiduk
>Assignee: niuyulin
>Priority: Minor
>
> Most places we log {{pid}} and {{ppid}} I noticed at least one occurrence of
> {{parent pid}} instead of {{ppid}}. All this logging should be uniform so
> that it's easy to grep the logs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-24669) Logging of ppid should be consistent across all occurrences
[
https://issues.apache.org/jira/browse/HBASE-24669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niuyulin reassigned HBASE-24669:
Assignee: niuyulin
> Logging of ppid should be consistent across all occurrences
> ---
>
> Key: HBASE-24669
> URL: https://issues.apache.org/jira/browse/HBASE-24669
> Project: HBase
> Issue Type: Improvement
> Components: Operability, proc-v2
>Reporter: Nick Dimiduk
>Assignee: niuyulin
>Priority: Minor
>
> Most places we log {{pid}} and {{ppid}} I noticed at least one occurrence of
> {{parent pid}} instead of {{ppid}}. All this logging should be uniform so
> that it's easy to grep the logs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-24659) Calcuate FIXED_OVERHEAD automatically
[ https://issues.apache.org/jira/browse/HBASE-24659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin reassigned HBASE-24659: Assignee: niuyulin > Calcuate FIXED_OVERHEAD automatically > - > > Key: HBASE-24659 > URL: https://issues.apache.org/jira/browse/HBASE-24659 > Project: HBase > Issue Type: Improvement >Reporter: Duo Zhang >Assignee: niuyulin >Priority: Major > > Now the FIXED_OVERHEAD in some classes are maintained manually, an we have a > method to TestHeapSizes to confirm that the value is correct. > But it is really hard for developers to count the fields in a complicated > class like HRegion. Since we have the ability to calcuate the accurate size > in UT, I think we it is also possible to calcuate it when loading the class, > which is a one time operation so should not effect the performance too much. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24669) Logging of ppid should be consistent across all occurrences
[
https://issues.apache.org/jira/browse/HBASE-24669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149782#comment-17149782
]
niuyulin commented on HBASE-24669:
--
you mean replace "parent pid" to "parent ppid"? what if parent has parent ?
{code:java}
2020-07-02,01:05:32,181 INFO [PEWorker-8]
org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished subprocedure
pid=6190915, resume processing parent pid=6190914, ppid=6190913,
state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED, locked=true;
TransitRegionStateProcedure table=tsdb,
region=15675415356879e7020f3ede695e0803, UNASSIGN
{code}
> Logging of ppid should be consistent across all occurrences
> ---
>
> Key: HBASE-24669
> URL: https://issues.apache.org/jira/browse/HBASE-24669
> Project: HBase
> Issue Type: Improvement
> Components: Operability, proc-v2
>Reporter: Nick Dimiduk
>Priority: Minor
>
> Most places we log {{pid}} and {{ppid}} I noticed at least one occurrence of
> {{parent pid}} instead of {{ppid}}. All this logging should be uniform so
> that it's easy to grep the logs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-24653) Show snapshot owner on Master WebUI
[ https://issues.apache.org/jira/browse/HBASE-24653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin reassigned HBASE-24653: Assignee: niuyulin > Show snapshot owner on Master WebUI > --- > > Key: HBASE-24653 > URL: https://issues.apache.org/jira/browse/HBASE-24653 > Project: HBase > Issue Type: Improvement >Reporter: Yi Mei >Assignee: niuyulin >Priority: Major > > Now Master UI shows lots of snapshot informations, and owner is also useful > to find out who create this snapshot. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24634) Promote TestLogRollingNoCluster to LargeTests
[ https://issues.apache.org/jira/browse/HBASE-24634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146414#comment-17146414 ] niuyulin commented on HBASE-24634: -- Yes, please help confirm it . > Promote TestLogRollingNoCluster to LargeTests > - > > Key: HBASE-24634 > URL: https://issues.apache.org/jira/browse/HBASE-24634 > Project: HBase > Issue Type: Task > Components: test >Reporter: Duo Zhang >Assignee: niuyulin >Priority: Major > Labels: beginner, trivial > Attachments: image-2020-06-26-00-26-51-151.png > > > It spends 7 minutes to finish on my local machine, so I think it should be a > LargeTests instead of MediumTests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24634) Promote TestLogRollingNoCluster to LargeTests
[ https://issues.apache.org/jira/browse/HBASE-24634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17145073#comment-17145073 ] niuyulin commented on HBASE-24634: -- !image-2020-06-26-00-26-51-151.png|width=794,height=185! > Promote TestLogRollingNoCluster to LargeTests > - > > Key: HBASE-24634 > URL: https://issues.apache.org/jira/browse/HBASE-24634 > Project: HBase > Issue Type: Task > Components: test >Reporter: Duo Zhang >Assignee: niuyulin >Priority: Major > Labels: beginner, trivial > Attachments: image-2020-06-26-00-26-51-151.png > > > It spends 7 minutes to finish on my local machine, so I think it should be a > LargeTests instead of MediumTests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24634) Promote TestLogRollingNoCluster to LargeTests
[ https://issues.apache.org/jira/browse/HBASE-24634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin updated HBASE-24634: - Attachment: image-2020-06-26-00-26-51-151.png > Promote TestLogRollingNoCluster to LargeTests > - > > Key: HBASE-24634 > URL: https://issues.apache.org/jira/browse/HBASE-24634 > Project: HBase > Issue Type: Task > Components: test >Reporter: Duo Zhang >Assignee: niuyulin >Priority: Major > Labels: beginner, trivial > Attachments: image-2020-06-26-00-26-51-151.png > > > It spends 7 minutes to finish on my local machine, so I think it should be a > LargeTests instead of MediumTests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24634) Promote TestLogRollingNoCluster to LargeTests
[ https://issues.apache.org/jira/browse/HBASE-24634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17145069#comment-17145069 ] niuyulin commented on HBASE-24634: -- only speed 30s on my local machine and almost the same time at jenkins ? > Promote TestLogRollingNoCluster to LargeTests > - > > Key: HBASE-24634 > URL: https://issues.apache.org/jira/browse/HBASE-24634 > Project: HBase > Issue Type: Task > Components: test >Reporter: Duo Zhang >Assignee: niuyulin >Priority: Major > Labels: beginner, trivial > > It spends 7 minutes to finish on my local machine, so I think it should be a > LargeTests instead of MediumTests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-24634) Promote TestLogRollingNoCluster to LargeTests
[ https://issues.apache.org/jira/browse/HBASE-24634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin reassigned HBASE-24634: Assignee: niuyulin > Promote TestLogRollingNoCluster to LargeTests > - > > Key: HBASE-24634 > URL: https://issues.apache.org/jira/browse/HBASE-24634 > Project: HBase > Issue Type: Task > Components: test >Reporter: Duo Zhang >Assignee: niuyulin >Priority: Major > Labels: beginner, trivial > > It spends 7 minutes to finish on my local machine, so I think it should be a > LargeTests instead of MediumTests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-24604) Remove the stable-1 notice on our download page
[ https://issues.apache.org/jira/browse/HBASE-24604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin reassigned HBASE-24604: Assignee: niuyulin > Remove the stable-1 notice on our download page > --- > > Key: HBASE-24604 > URL: https://issues.apache.org/jira/browse/HBASE-24604 > Project: HBase > Issue Type: Task >Reporter: Duo Zhang >Assignee: niuyulin >Priority: Major > > We have already removed it from our dist release directory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24508) Why ProtobufUtil does not set scan's limit
[
https://issues.apache.org/jira/browse/HBASE-24508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17127219#comment-17127219
]
niuyulin commented on HBASE-24508:
--
ScanRequest has limit, we use this one for calls
> Why ProtobufUtil does not set scan's limit
>
>
> Key: HBASE-24508
> URL: https://issues.apache.org/jira/browse/HBASE-24508
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.2.5
>Reporter: yukunpeng
>Priority: Trivial
>
> {code:java}
> //ProtobufUtil
> /**
> * Convert a client Scan to a protocol buffer Scan
> *
> * @param scan the client Scan to convert
> * @return the converted protocol buffer Scan
> * @throws IOException
> */
> public static ClientProtos.Scan toScan(
> final Scan scan) throws IOException {
> ClientProtos.Scan.Builder scanBuilder =
> ClientProtos.Scan.newBuilder();
> scanBuilder.setCacheBlocks(scan.getCacheBlocks());
> if (scan.getBatch() > 0) {
> scanBuilder.setBatchSize(scan.getBatch());
> }
> if (scan.getMaxResultSize() > 0) {
> scanBuilder.setMaxResultSize(scan.getMaxResultSize());
> }
> if (scan.isSmall()) {
> scanBuilder.setSmall(scan.isSmall());
> }
> if (scan.getAllowPartialResults()) {
> scanBuilder.setAllowPartialResults(scan.getAllowPartialResults());
> }
> Boolean loadColumnFamiliesOnDemand =
> scan.getLoadColumnFamiliesOnDemandValue();
> if (loadColumnFamiliesOnDemand != null) {
> scanBuilder.setLoadColumnFamiliesOnDemand(loadColumnFamiliesOnDemand);
> }
> scanBuilder.setMaxVersions(scan.getMaxVersions());
> scan.getColumnFamilyTimeRange().forEach((cf, timeRange) -> {
> scanBuilder.addCfTimeRange(HBaseProtos.ColumnFamilyTimeRange.newBuilder()
> .setColumnFamily(UnsafeByteOperations.unsafeWrap(cf))
> .setTimeRange(toTimeRange(timeRange))
> .build());
> });
> scanBuilder.setTimeRange(ProtobufUtil.toTimeRange(scan.getTimeRange()));
> Map attributes = scan.getAttributesMap();
> if (!attributes.isEmpty()) {
> NameBytesPair.Builder attributeBuilder = NameBytesPair.newBuilder();
> for (Map.Entry attribute: attributes.entrySet()) {
> attributeBuilder.setName(attribute.getKey());
>
> attributeBuilder.setValue(UnsafeByteOperations.unsafeWrap(attribute.getValue()));
> scanBuilder.addAttribute(attributeBuilder.build());
> }
> }
> byte[] startRow = scan.getStartRow();
> if (startRow != null && startRow.length > 0) {
> scanBuilder.setStartRow(UnsafeByteOperations.unsafeWrap(startRow));
> }
> byte[] stopRow = scan.getStopRow();
> if (stopRow != null && stopRow.length > 0) {
> scanBuilder.setStopRow(UnsafeByteOperations.unsafeWrap(stopRow));
> }
> if (scan.hasFilter()) {
> scanBuilder.setFilter(ProtobufUtil.toFilter(scan.getFilter()));
> }
> if (scan.hasFamilies()) {
> Column.Builder columnBuilder = Column.newBuilder();
> for (Map.Entry>
> family: scan.getFamilyMap().entrySet()) {
>
> columnBuilder.setFamily(UnsafeByteOperations.unsafeWrap(family.getKey()));
> NavigableSet qualifiers = family.getValue();
> columnBuilder.clearQualifier();
> if (qualifiers != null && qualifiers.size() > 0) {
> for (byte [] qualifier: qualifiers) {
>
> columnBuilder.addQualifier(UnsafeByteOperations.unsafeWrap(qualifier));
> }
> }
> scanBuilder.addColumn(columnBuilder.build());
> }
> }
> if (scan.getMaxResultsPerColumnFamily() >= 0) {
> scanBuilder.setStoreLimit(scan.getMaxResultsPerColumnFamily());
> }
> if (scan.getRowOffsetPerColumnFamily() > 0) {
> scanBuilder.setStoreOffset(scan.getRowOffsetPerColumnFamily());
> }
> if (scan.isReversed()) {
> scanBuilder.setReversed(scan.isReversed());
> }
> if (scan.getConsistency() == Consistency.TIMELINE) {
> scanBuilder.setConsistency(toConsistency(scan.getConsistency()));
> }
> if (scan.getCaching() > 0) {
> scanBuilder.setCaching(scan.getCaching());
> }
> long mvccReadPoint = PackagePrivateFieldAccessor.getMvccReadPoint(scan);
> if (mvccReadPoint > 0) {
> scanBuilder.setMvccReadPoint(mvccReadPoint);
> }
> if (!scan.includeStartRow()) {
> scanBuilder.setIncludeStartRow(false);
> }
> scanBuilder.setIncludeStopRow(scan.includeStopRow());
> if (scan.getReadType() != Scan.ReadType.DEFAULT) {
> scanBuilder.setReadType(toReadType(scan.getReadType()));
> }
> if (scan.isNeedCursorResult()) {
> scanBuilder.setNeedCursorResult(true);
> }
> return scanBuilder.build();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24412) Canary support check only one column family per RegionTask
[ https://issues.apache.org/jira/browse/HBASE-24412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin updated HBASE-24412: - Fix Version/s: 2.2.6 2.3.0 3.0.0-alpha-1 > Canary support check only one column family per RegionTask > -- > > Key: HBASE-24412 > URL: https://issues.apache.org/jira/browse/HBASE-24412 > Project: HBase > Issue Type: Improvement > Components: canary >Reporter: niuyulin >Assignee: niuyulin >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0, 2.2.6 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24412) Canary support check only one column family per RegionTask
[ https://issues.apache.org/jira/browse/HBASE-24412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin resolved HBASE-24412. -- Resolution: Fixed > Canary support check only one column family per RegionTask > -- > > Key: HBASE-24412 > URL: https://issues.apache.org/jira/browse/HBASE-24412 > Project: HBase > Issue Type: Improvement > Components: canary >Reporter: niuyulin >Assignee: niuyulin >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0, 2.2.6 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24504) refactor call setupCluster/tearDownCluster in TestTableSnapshotInputFormat
niuyulin created HBASE-24504: Summary: refactor call setupCluster/tearDownCluster in TestTableSnapshotInputFormat Key: HBASE-24504 URL: https://issues.apache.org/jira/browse/HBASE-24504 Project: HBase Issue Type: Improvement Components: mapreduce, test Reporter: niuyulin Assignee: niuyulin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24412) Canary support check only one column family per RegionTask
[ https://issues.apache.org/jira/browse/HBASE-24412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124538#comment-17124538 ] niuyulin commented on HBASE-24412: -- [~claraxiong] 1.Could you provide some info on the use case to support checking all families and a random family? -->Reduce side effects of canary sniff on hdd cluster with large amount regions 2.Could you expose the new option in usage? -->you can config "hbase.canary.read.all.column.famliy" 3.It was nice to clean up the old code by reformatting. But this makes review difficult.Could you undo formatting and keep formatting as the last commit? -->jenkins will check PR, we'd better ensure no "-1" before merge, including code format error > Canary support check only one column family per RegionTask > -- > > Key: HBASE-24412 > URL: https://issues.apache.org/jira/browse/HBASE-24412 > Project: HBase > Issue Type: Improvement > Components: canary >Reporter: niuyulin >Assignee: niuyulin >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24412) Canary support check only one column family per RegionTask
[ https://issues.apache.org/jira/browse/HBASE-24412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin updated HBASE-24412: - Summary: Canary support check only one column family per RegionTask (was: Canary support check only one column family per region) > Canary support check only one column family per RegionTask > -- > > Key: HBASE-24412 > URL: https://issues.apache.org/jira/browse/HBASE-24412 > Project: HBase > Issue Type: Improvement > Components: canary >Reporter: niuyulin >Assignee: niuyulin >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24450) There was a partial failure due to IO when attempting to load
[ https://issues.apache.org/jira/browse/HBASE-24450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118282#comment-17118282 ] niuyulin commented on HBASE-24450: -- client side will retry, we have config "hbase.bulkload.retries.number" > There was a partial failure due to IO when attempting to load > - > > Key: HBASE-24450 > URL: https://issues.apache.org/jira/browse/HBASE-24450 > Project: HBase > Issue Type: Bug >Affects Versions: 2.2.4 >Reporter: yukunpeng >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24412) Canary support check only one column family per region
niuyulin created HBASE-24412: Summary: Canary support check only one column family per region Key: HBASE-24412 URL: https://issues.apache.org/jira/browse/HBASE-24412 Project: HBase Issue Type: Improvement Components: canary Reporter: niuyulin Assignee: niuyulin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24387) TableSnapshotInputFormatImpl support row limit on each InputSplit
[ https://issues.apache.org/jira/browse/HBASE-24387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin updated HBASE-24387: - Summary: TableSnapshotInputFormatImpl support row limit on each InputSplit (was: TableSnapshotInputFormatImpl support scan limit on each InputSplit) > TableSnapshotInputFormatImpl support row limit on each InputSplit > - > > Key: HBASE-24387 > URL: https://issues.apache.org/jira/browse/HBASE-24387 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: niuyulin >Assignee: niuyulin >Priority: Major > > In some scenario , We want to scan limited rows on each InputSplit for > sampling data extraction -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24387) TableSnapshotInputFormatImpl support scan limit on each InputSplit
[ https://issues.apache.org/jira/browse/HBASE-24387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin updated HBASE-24387: - Description: In some scenario , We want to scan limited rows on each InputSplit for sampling data extraction (was: In some scenario , We want to scan limited rows on each InputSplit for sampling verification) > TableSnapshotInputFormatImpl support scan limit on each InputSplit > -- > > Key: HBASE-24387 > URL: https://issues.apache.org/jira/browse/HBASE-24387 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: niuyulin >Assignee: niuyulin >Priority: Major > > In some scenario , We want to scan limited rows on each InputSplit for > sampling data extraction -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24387) TableSnapshotInputFormatImpl support scan limit on each InputSplit
[ https://issues.apache.org/jira/browse/HBASE-24387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin updated HBASE-24387: - Description: In some scenario , We want to scan limited rows on each InputSplit for sampling verification > TableSnapshotInputFormatImpl support scan limit on each InputSplit > -- > > Key: HBASE-24387 > URL: https://issues.apache.org/jira/browse/HBASE-24387 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Reporter: niuyulin >Assignee: niuyulin >Priority: Major > > In some scenario , We want to scan limited rows on each InputSplit for > sampling verification -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24387) TableSnapshotInputFormatImpl support scan limit on each InputSplit
niuyulin created HBASE-24387: Summary: TableSnapshotInputFormatImpl support scan limit on each InputSplit Key: HBASE-24387 URL: https://issues.apache.org/jira/browse/HBASE-24387 Project: HBase Issue Type: Improvement Components: mapreduce Reporter: niuyulin Assignee: niuyulin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24386) TableSnapshotScanner support scan limit
niuyulin created HBASE-24386: Summary: TableSnapshotScanner support scan limit Key: HBASE-24386 URL: https://issues.apache.org/jira/browse/HBASE-24386 Project: HBase Issue Type: Improvement Components: Scanners, snapshots Reporter: niuyulin Assignee: niuyulin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24328) skip duplicate GCMultipleMergedRegionsProcedure while previous finished
[ https://issues.apache.org/jira/browse/HBASE-24328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin resolved HBASE-24328. -- Resolution: Fixed > skip duplicate GCMultipleMergedRegionsProcedure while previous finished > --- > > Key: HBASE-24328 > URL: https://issues.apache.org/jira/browse/HBASE-24328 > Project: HBase > Issue Type: Improvement >Reporter: niuyulin >Assignee: niuyulin >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24328) skip duplicate GCMultipleMergedRegionsProcedure while previous finished
niuyulin created HBASE-24328: Summary: skip duplicate GCMultipleMergedRegionsProcedure while previous finished Key: HBASE-24328 URL: https://issues.apache.org/jira/browse/HBASE-24328 Project: HBase Issue Type: Improvement Reporter: niuyulin Assignee: niuyulin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24250) CatalogJanitor resubmits GCMultipleMergedRegionsProcedure for the same region
[ https://issues.apache.org/jira/browse/HBASE-24250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099795#comment-17099795 ] niuyulin commented on HBASE-24250: -- [~huaxiangsun] yes, You are right! The deleteMergeQualifiers method should be idempotent ,but actually not . the second run makes delete Columns into delete row , so it make hole > CatalogJanitor resubmits GCMultipleMergedRegionsProcedure for the same region > - > > Key: HBASE-24250 > URL: https://issues.apache.org/jira/browse/HBASE-24250 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 2.2.4 > Environment: hdfs 3.1.3 with erasure coding > hbase 2.2.4 >Reporter: Andrey Elenskiy >Assignee: niuyulin >Priority: Major > > If a lot of regions were merged (due to change of region sizes, for example), > there can be a long backlog of procedures to clean up the merged regions. If > going through this backlog is slower than the CatalogJanitor's scan interval, > it will end resubmitting GCMultipleMergedRegionsProcedure for the same > regions over and over again. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-22069) fix the download resource absences from Apache urls by scripts automatically
[
https://issues.apache.org/jira/browse/HBASE-22069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
niuyulin reassigned HBASE-22069:
Assignee: (was: niuyulin)
> fix the download resource absences from Apache urls by scripts automatically
>
>
> Key: HBASE-22069
> URL: https://issues.apache.org/jira/browse/HBASE-22069
> Project: HBase
> Issue Type: Sub-task
>Reporter: Junhong Xu
>Priority: Major
>
> Just as [~elserj] said in HBASE-22038 ,
> {quote}archive.a.o should not be used by build processes. www-us.apache.org
> is a load-balanced endpoint, whereas archive.a.o is a single service.We
> should just switch to newer versions of these libraries that still exist on
> www-us.apache.org. Would you be able to try that change out in a follow-on?
> {quote}
>
> And as the comment from [~busbey] there has been a script named
> dev-support/jenkins-scripts/cache-apache-project-artifact.sh that fix the
> problem as the title. It is uniform to fix resource absences from the
> apache-us url by the script above.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-24255) GCRegionProcedure doesn't assign region from RegionServer leading to orphans
[ https://issues.apache.org/jira/browse/HBASE-24255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] niuyulin reassigned HBASE-24255: Assignee: (was: niuyulin) > GCRegionProcedure doesn't assign region from RegionServer leading to orphans > > > Key: HBASE-24255 > URL: https://issues.apache.org/jira/browse/HBASE-24255 > Project: HBase > Issue Type: Bug > Components: proc-v2, Region Assignment, regionserver >Affects Versions: 2.2.4 > Environment: hbase 2.2.4 > hadoop 3.1.3 >Reporter: Andrey Elenskiy >Priority: Major > > We've found ourselves in a situation where parents of merged or split regions > needed to be opened again on a regionserver due to having to recover from > cluster meltdown (HBCK2's fixMeta kicks off GCMultipleMergedRegionsProcedure > which requiters all regions to be merged to be open). Then, when a > GCProcedure is kicked of to clean a parent region up by > GCMultipleMergedRegionsProcedure, it ends up deleting it from hbase:meta, but > doesn't unassign it from RegionServer leading for it to show up in "Orphan > Regions on RegionServer" in hbck tab of HBase Master. Also, the hbase client > doesn't detect that the region is closed either because it's still > technically open on a regionserver (it doesn't reread hbase:meta all the > time). The only way to recover from this is to restart regionserver which > isn't idea as it can lead to other issues in clusters with region > inconsistencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24221) Support bulkLoadHFile by family
[ https://issues.apache.org/jira/browse/HBASE-24221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098452#comment-17098452 ] niuyulin commented on HBASE-24221: -- Hi [~wchevreuil] , I have submit another PR > Support bulkLoadHFile by family > --- > > Key: HBASE-24221 > URL: https://issues.apache.org/jira/browse/HBASE-24221 > Project: HBase > Issue Type: Improvement > Components: HFile >Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.2.4 >Reporter: niuyulin >Assignee: niuyulin >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0 > > > Support bulkLoadHFile by family to avoid long time waiting of bulkLoadHFile > because of compacting at server side -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24250) CatalogJanitor resubmits GCMultipleMergedRegionsProcedure for the same region
[ https://issues.apache.org/jira/browse/HBASE-24250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097802#comment-17097802 ] niuyulin edited comment on HBASE-24250 at 5/2/20, 2:36 PM: --- [~huaxiangsun] Maybe you forget add the Procedures Last round generate into ongoingMergeProcedures? you cannot sure it finish in last round. it can work, but when GCMultipleMergedRegionsProcedure finish or failed exit, you must clean the Procedure in ongoingMergeProcedures was (Author: niuyulin): [~huaxiangsun] Maybe you forget add the Procedures Last round generate into ongoingMergeProcedures, you cannot sure it finish in last round. it can work, but when GCMultipleMergedRegionsProcedure finish or failed exit, you must clean the Procedure in ongoingMergeProcedures > CatalogJanitor resubmits GCMultipleMergedRegionsProcedure for the same region > - > > Key: HBASE-24250 > URL: https://issues.apache.org/jira/browse/HBASE-24250 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 2.2.4 > Environment: hdfs 3.1.3 with erasure coding > hbase 2.2.4 >Reporter: Andrey Elenskiy >Assignee: niuyulin >Priority: Major > > If a lot of regions were merged (due to change of region sizes, for example), > there can be a long backlog of procedures to clean up the merged regions. If > going through this backlog is slower than the CatalogJanitor's scan interval, > it will end resubmitting GCMultipleMergedRegionsProcedure for the same > regions over and over again. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24250) CatalogJanitor resubmits GCMultipleMergedRegionsProcedure for the same region
[ https://issues.apache.org/jira/browse/HBASE-24250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097990#comment-17097990 ] niuyulin commented on HBASE-24250: -- [~huaxiangsun] , I submit PR [https://github.com/apache/hbase/pull/1629] I do it in a simple way, could you help review it > CatalogJanitor resubmits GCMultipleMergedRegionsProcedure for the same region > - > > Key: HBASE-24250 > URL: https://issues.apache.org/jira/browse/HBASE-24250 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 2.2.4 > Environment: hdfs 3.1.3 with erasure coding > hbase 2.2.4 >Reporter: Andrey Elenskiy >Assignee: niuyulin >Priority: Major > > If a lot of regions were merged (due to change of region sizes, for example), > there can be a long backlog of procedures to clean up the merged regions. If > going through this backlog is slower than the CatalogJanitor's scan interval, > it will end resubmitting GCMultipleMergedRegionsProcedure for the same > regions over and over again. -- This message was sent by Atlassian Jira (v8.3.4#803005)
