[jira] [Commented] (HBASE-19450) Add log about average execution time for ScheduledChore
[ https://issues.apache.org/jira/browse/HBASE-19450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975596#comment-16975596 ] Sean Busbey commented on HBASE-19450: - no we don't log both. we only ever log one. it's just that at trace level you see TRACE message after each invocation and then at the 5 minute mark you'd see one at INFO instead of at TRACE. > Add log about average execution time for ScheduledChore > --- > > Key: HBASE-19450 > URL: https://issues.apache.org/jira/browse/HBASE-19450 > Project: HBase > Issue Type: Improvement > Components: Operability >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Minor > Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3 > > Attachments: HBASE-19450.addendum1.0.patch, > HBASE-19450.master.001.patch, HBASE-19450.master.002.patch, > HBASE-19450.master.003.patch, HBASE-19450.master.004.patch > > > So far, there is no information about the exact execution time for a chore, > we can provide log information about it. It also brings other benefits, like > discovering inefficient chores which show rooms for improvement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-19450) Add log about average execution time for ScheduledChore
[ https://issues.apache.org/jira/browse/HBASE-19450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975591#comment-16975591 ] Sean Busbey commented on HBASE-19450: - > On the patch, we wrap LOG.trace with LOG.isInfoEnabled? Seems not good. Please read the full context. We have to do some work to decide to log at info or at trace. If we aren't at least at info level we skip doing all the work. > Add log about average execution time for ScheduledChore > --- > > Key: HBASE-19450 > URL: https://issues.apache.org/jira/browse/HBASE-19450 > Project: HBase > Issue Type: Improvement > Components: Operability >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Minor > Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3 > > Attachments: HBASE-19450.addendum1.0.patch, > HBASE-19450.master.001.patch, HBASE-19450.master.002.patch, > HBASE-19450.master.003.patch, HBASE-19450.master.004.patch > > > So far, there is no information about the exact execution time for a chore, > we can provide log information about it. It also brings other benefits, like > discovering inefficient chores which show rooms for improvement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-19450) Add log about average execution time for ScheduledChore
[ https://issues.apache.org/jira/browse/HBASE-19450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975590#comment-16975590 ] Sean Busbey commented on HBASE-19450: - It has not been resolved for two years. It got resolved earlier today. We should not release with the original commit without the addendum. As stack mentioned we log at least once a second > Add log about average execution time for ScheduledChore > --- > > Key: HBASE-19450 > URL: https://issues.apache.org/jira/browse/HBASE-19450 > Project: HBase > Issue Type: Improvement > Components: Operability >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Minor > Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3 > > Attachments: HBASE-19450.addendum1.0.patch, > HBASE-19450.master.001.patch, HBASE-19450.master.002.patch, > HBASE-19450.master.003.patch, HBASE-19450.master.004.patch > > > So far, there is no information about the exact execution time for a chore, > we can provide log information about it. It also brings other benefits, like > discovering inefficient chores which show rooms for improvement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23302) Put up 2.1.8RC0
[ https://issues.apache.org/jira/browse/HBASE-23302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975575#comment-16975575 ] Sean Busbey commented on HBASE-23302: - you should wait for the addendum on HBASE-19450 to land before making the RC. it's waiting for a review > Put up 2.1.8RC0 > --- > > Key: HBASE-23302 > URL: https://issues.apache.org/jira/browse/HBASE-23302 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-19450) Add log about average execution time for ScheduledChore
[ https://issues.apache.org/jira/browse/HBASE-19450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975565#comment-16975565 ] Sean Busbey commented on HBASE-19450: - I think the addendum works well enough. precommit gives it a pass. anyone got a +1 for it? > Add log about average execution time for ScheduledChore > --- > > Key: HBASE-19450 > URL: https://issues.apache.org/jira/browse/HBASE-19450 > Project: HBase > Issue Type: Improvement > Components: Operability >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Minor > Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3 > > Attachments: HBASE-19450.addendum1.0.patch, > HBASE-19450.master.001.patch, HBASE-19450.master.002.patch, > HBASE-19450.master.003.patch, HBASE-19450.master.004.patch > > > So far, there is no information about the exact execution time for a chore, > we can provide log information about it. It also brings other benefits, like > discovering inefficient chores which show rooms for improvement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-19450) Add log about average execution time for ScheduledChore
[ https://issues.apache.org/jira/browse/HBASE-19450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-19450: Status: Patch Available (was: Reopened) addendum v0 - Ensure we don't output the execution time at INFO more often then once per 5 minutes - when not logging at INFO, log at TRACE Tested by running a stand alone instance and going through logs out of the box, and then runtime changing the log level via the web ui on ScheduledChore to TRACE, to WARN, and back to INFO. > Add log about average execution time for ScheduledChore > --- > > Key: HBASE-19450 > URL: https://issues.apache.org/jira/browse/HBASE-19450 > Project: HBase > Issue Type: Improvement > Components: Operability >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Minor > Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3 > > Attachments: HBASE-19450.addendum1.0.patch, > HBASE-19450.master.001.patch, HBASE-19450.master.002.patch, > HBASE-19450.master.003.patch, HBASE-19450.master.004.patch > > > So far, there is no information about the exact execution time for a chore, > we can provide log information about it. It also brings other benefits, like > discovering inefficient chores which show rooms for improvement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-19450) Add log about average execution time for ScheduledChore
[ https://issues.apache.org/jira/browse/HBASE-19450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-19450: Attachment: HBASE-19450.addendum1.0.patch > Add log about average execution time for ScheduledChore > --- > > Key: HBASE-19450 > URL: https://issues.apache.org/jira/browse/HBASE-19450 > Project: HBase > Issue Type: Improvement > Components: Operability >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Minor > Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3 > > Attachments: HBASE-19450.addendum1.0.patch, > HBASE-19450.master.001.patch, HBASE-19450.master.002.patch, > HBASE-19450.master.003.patch, HBASE-19450.master.004.patch > > > So far, there is no information about the exact execution time for a chore, > we can provide log information about it. It also brings other benefits, like > discovering inefficient chores which show rooms for improvement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-19450) Add log about average execution time for ScheduledChore
[ https://issues.apache.org/jira/browse/HBASE-19450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975531#comment-16975531 ] Sean Busbey commented on HBASE-19450: - I think I got this going. let me get an addendum up > Add log about average execution time for ScheduledChore > --- > > Key: HBASE-19450 > URL: https://issues.apache.org/jira/browse/HBASE-19450 > Project: HBase > Issue Type: Improvement > Components: Operability >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Minor > Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3 > > Attachments: HBASE-19450.master.001.patch, > HBASE-19450.master.002.patch, HBASE-19450.master.003.patch, > HBASE-19450.master.004.patch > > > So far, there is no information about the exact execution time for a chore, > we can provide log information about it. It also brings other benefits, like > discovering inefficient chores which show rooms for improvement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-19450) Add log about average execution time for ScheduledChore
[ https://issues.apache.org/jira/browse/HBASE-19450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975462#comment-16975462 ] Sean Busbey commented on HBASE-19450: - Huh. Those are very short chore periods. What do you think, set a minimum logging period instead of each time? Maybe at most once per 5 minutes? > Add log about average execution time for ScheduledChore > --- > > Key: HBASE-19450 > URL: https://issues.apache.org/jira/browse/HBASE-19450 > Project: HBase > Issue Type: Improvement > Components: Operability >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Minor > Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3 > > Attachments: HBASE-19450.master.001.patch, > HBASE-19450.master.002.patch, HBASE-19450.master.003.patch, > HBASE-19450.master.004.patch > > > So far, there is no information about the exact execution time for a chore, > we can provide log information about it. It also brings other benefits, like > discovering inefficient chores which show rooms for improvement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-19450) Add log about average execution time for ScheduledChore
[ https://issues.apache.org/jira/browse/HBASE-19450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16975445#comment-16975445 ] Sean Busbey commented on HBASE-19450: - I went through minicluster logs and it seemed fine. You running at INFO or DEBUG? > Add log about average execution time for ScheduledChore > --- > > Key: HBASE-19450 > URL: https://issues.apache.org/jira/browse/HBASE-19450 > Project: HBase > Issue Type: Improvement > Components: Operability >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Minor > Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3 > > Attachments: HBASE-19450.master.001.patch, > HBASE-19450.master.002.patch, HBASE-19450.master.003.patch, > HBASE-19450.master.004.patch > > > So far, there is no information about the exact execution time for a chore, > we can provide log information about it. It also brings other benefits, like > discovering inefficient chores which show rooms for improvement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23287) LogCleaner is not added to choreService
[
https://issues.apache.org/jira/browse/HBASE-23287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-23287:
Fix Version/s: 1.4.12
1.6.0
Release Note:
Prior to this change HBase 1.4.11 and 1.5.0 based clusters would fail to clean
up WALs from the oldWALs area automatically. With this change in place clusters
should return to the expected behavior of cleaning up WALs that are no longer
needed.
Resolution: Fixed
Status: Resolved (was: Patch Available)
> LogCleaner is not added to choreService
> ---
>
> Key: HBASE-23287
> URL: https://issues.apache.org/jira/browse/HBASE-23287
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.6.0, 1.4.11
>Reporter: Baiqiang Zhao
>Assignee: Baiqiang Zhao
>Priority: Blocker
> Fix For: 1.6.0, 1.4.12
>
> Attachments: HBASE-23287.branch-1.0001.patch
>
>
> We found /hbase/oldWALs keeps growing, and we never use replication feature.
> Then we found OldWALsCleaner thread is always in WAITTING state :
> {code:java}
> Thread 176 (OldWALsCleaner-1): State: WAITING Blocked count: 0 Waited count:
> 1 Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2ef05d36
> Stack: sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> org.apache.hadoop.hbase.master.cleaner.LogCleaner.deleteFile(LogCleaner.java:185)
>
> org.apache.hadoop.hbase.master.cleaner.LogCleaner.access$000(LogCleaner.java:48)
> org.apache.hadoop.hbase.master.cleaner.LogCleaner$1.run(LogCleaner.java:161)
> java.lang.Thread.run(Thread.java:745)
> {code}
> Finally, we found that LogCleaner is not added to choreService, so LogCleaner
> doesn't run periodically.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-22701) Better handle invalid local directory for DynamicClassLoader
[
https://issues.apache.org/jira/browse/HBASE-22701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-22701:
Fix Version/s: 1.4.12
1.6.0
> Better handle invalid local directory for DynamicClassLoader
>
>
> Key: HBASE-22701
> URL: https://issues.apache.org/jira/browse/HBASE-22701
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 2.3.0, 1.6.0, 2.1.7, 2.2.2, 1.4.12
>
> Attachments: HBASE-22701.001.patch, HBASE-22701.002.patch,
> HBASE-22701.003.patch
>
>
> If you give HBase an {{hbase.local.dir}} (usually, "{{hbase.tmp.dir}}/local")
> which is not writable to it, you will get some weird errors on the scan path.
> I just saw this (again?) with Phoenix.
> Specifically, the first attempt to reference DynamicClassLoader (via
> ProtobufUtil), will result in an ExceptionInInitializationError because the
> unchecked exception coming out of DynamicClassLoader's constructor interrupts
> the loading of {{DynamicClassLoader.class}}.
> {noformat}
> 2019-07-14 06:25:34,284 ERROR
> [RpcServer.Metadata.Fifo.handler=12,queue=0,port=16020]
> coprocessor.MetaDataEndpointImpl: dropTable failed
> org.apache.hadoop.hbase.DoNotRetryIOException:
> java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.translateException(RpcRetryingCallerImpl.java:221)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:194)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:387)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:361)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
> at
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1598)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toScan(ProtobufUtil.java:1152)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2967)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3301)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:332)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:242)
> at
> org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58)
> at
> org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:127)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192)
> ... 7 more
> Caused by: java.lang.RuntimeException: Failed to create local dir
> /hadoopfs/fs1/hbase/local/jars, DynamicClassLoader failed to init
> at
> org.apache.hadoop.hbase.util.DynamicClassLoader.initTempDir(DynamicClassLoader.java:110)
> at
> org.apache.hadoop.hbase.util.DynamicClassLoader.(DynamicClassLoader.java:98)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder.lambda$static$0(ProtobufUtil.java:261)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder.(ProtobufUtil.java:260)
> ... 16 more
> {noformat}
> Every subsequent call will result in a NoClassDefFoundError, because we
> already tried to load DynamicClassLoader.class once and failed.
> {noformat}
> 2019-07-14 06:25:34,380 ERROR
> [RpcServer.Metadata.Fifo.handler=2,queue=2,port=16020]
> coprocessor.MetaDataEndpointImpl: dropTable failed
> org.apache.hadoop.hbase.DoNotRetryIOException:
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil$ClassLoaderHolder
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.translateException(RpcRetryingCallerImpl.java:221)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:194)
> at
>
[jira] [Commented] (HBASE-19450) Add log about average execution time for ScheduledChore
[ https://issues.apache.org/jira/browse/HBASE-19450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974770#comment-16974770 ] Sean Busbey commented on HBASE-19450: - This would have helped me this week, so I rebased it up and pushed it. Thanks [~reidchan]! > Add log about average execution time for ScheduledChore > --- > > Key: HBASE-19450 > URL: https://issues.apache.org/jira/browse/HBASE-19450 > Project: HBase > Issue Type: Improvement > Components: Operability >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Minor > Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3 > > Attachments: HBASE-19450.master.001.patch, > HBASE-19450.master.002.patch, HBASE-19450.master.003.patch, > HBASE-19450.master.004.patch > > > So far, there is no information about the exact execution time for a chore, > we can provide log information about it. It also brings other benefits, like > discovering inefficient chores which show rooms for improvement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-19450) Add log about average execution time for ScheduledChore
[ https://issues.apache.org/jira/browse/HBASE-19450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-19450: Fix Version/s: 2.2.3 2.1.8 2.3.0 3.0.0 Release Note: HBase internal chores now log a moving average of how long execution of each chore takes at `INFO` level for the logger `org.apache.hadoop.hbase.ScheduledChore`. Resolution: Fixed Status: Resolved (was: Patch Available) > Add log about average execution time for ScheduledChore > --- > > Key: HBASE-19450 > URL: https://issues.apache.org/jira/browse/HBASE-19450 > Project: HBase > Issue Type: Improvement > Components: Operability >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Minor > Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3 > > Attachments: HBASE-19450.master.001.patch, > HBASE-19450.master.002.patch, HBASE-19450.master.003.patch, > HBASE-19450.master.004.patch > > > So far, there is no information about the exact execution time for a chore, > we can provide log information about it. It also brings other benefits, like > discovering inefficient chores which show rooms for improvement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-19450) Add log about average execution time for ScheduledChore
[ https://issues.apache.org/jira/browse/HBASE-19450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-19450: Component/s: Operability > Add log about average execution time for ScheduledChore > --- > > Key: HBASE-19450 > URL: https://issues.apache.org/jira/browse/HBASE-19450 > Project: HBase > Issue Type: Improvement > Components: Operability >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Minor > Attachments: HBASE-19450.master.001.patch, > HBASE-19450.master.002.patch, HBASE-19450.master.003.patch, > HBASE-19450.master.004.patch > > > So far, there is no information about the exact execution time for a chore, > we can provide log information about it. It also brings other benefits, like > discovering inefficient chores which show rooms for improvement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23283) Provide clear and consistent logging about the period of enabled chores
[
https://issues.apache.org/jira/browse/HBASE-23283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-23283:
Fix Version/s: 2.2.3
2.1.8
1.6.0
2.3.0
3.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
pushed to all branches. Thanks!
> Provide clear and consistent logging about the period of enabled chores
> ---
>
> Key: HBASE-23283
> URL: https://issues.apache.org/jira/browse/HBASE-23283
> Project: HBase
> Issue Type: Improvement
> Components: Operability
>Affects Versions: 3.0.0, 2.3.0, 1.7.0
>Reporter: Sean Busbey
>Assignee: Mingliang Liu
>Priority: Minor
> Fix For: 3.0.0, 2.3.0, 1.6.0, 2.1.8, 2.2.3
>
> Attachments: HBASE-23283.000.patch
>
>
> Similar to HBASE-23038, we should always log info about our enabled chores.
> Right now wether or not we get some information is up to particular Chore
> constructors and by and large we don't get any log messages when things can
> get started, even if the period is something impossibly long (e.g. 3000 days).
> When we go to schedule the chore here:
> {code}
> if (chore.getPeriod() <= 0) {
> LOG.info("The period is {} seconds, {} is disabled",
> chore.getPeriod(), chore.getName());
> return false;
> }
> {code}
> we should add an else clause that says it's enabled. It looks like we could
> then just call chore.toString to get the proper details about the chore and
> its period.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23038) Provide consistent and clear logging about disabling chores
[
https://issues.apache.org/jira/browse/HBASE-23038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-23038:
Fix Version/s: 2.2.3
2.1.8
> Provide consistent and clear logging about disabling chores
> ---
>
> Key: HBASE-23038
> URL: https://issues.apache.org/jira/browse/HBASE-23038
> Project: HBase
> Issue Type: Improvement
> Components: master, regionserver
>Reporter: Sean Busbey
>Assignee: Sanjeet Nishad
>Priority: Minor
> Labels: beginner
> Fix For: 3.0.0, 1.5.0, 2.3.0, 2.1.8, 2.2.3
>
> Attachments: HBASE-23038.001.patch, HBASE-23038.002.patch,
> HBASE-23038.master.003.patch
>
>
> Right now if you want to disable any of our chores you can set the period to
> be <= 0. Sometimes, if you do this you get a nice message:
> {code}
> 2019-09-16 22:10:16,756 INFO [master-1:16000.activeMasterManager]
> master.HMaster: The period is 0 seconds, MobCompactionChore is disabled
> {code}
> And sometimes you get an opaque message:
> {code}
> 2019-09-16 22:09:45,333 INFO [master-1:16000.activeMasterManager]
> hbase.ChoreService: Could not successfully schedule chore: LogsCleaner
> 2019-09-16 22:09:45,340 INFO [master-1:16000.activeMasterManager]
> hbase.ChoreService: Could not successfully schedule chore: HFileCleaner
> {code}
> This is because sometimes we just blindly submit to ChoreService which
> submits to a java ScheduledExecutorService and then catches the
> IllegalArgumentException.
> We should remove the one-offs and make it so ChoreService checks the period
> before accepting a submittal and produces a consistent "Foo is disabled"
> message.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23283) Provide clear and consistent logging about the period of enabled chores
[
https://issues.apache.org/jira/browse/HBASE-23283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974717#comment-16974717
]
Sean Busbey commented on HBASE-23283:
-
No worries, I'll push it. Patience please.
> Provide clear and consistent logging about the period of enabled chores
> ---
>
> Key: HBASE-23283
> URL: https://issues.apache.org/jira/browse/HBASE-23283
> Project: HBase
> Issue Type: Improvement
> Components: Operability
>Affects Versions: 3.0.0, 2.3.0, 1.7.0
>Reporter: Sean Busbey
>Assignee: Mingliang Liu
>Priority: Minor
> Attachments: HBASE-23283.000.patch
>
>
> Similar to HBASE-23038, we should always log info about our enabled chores.
> Right now wether or not we get some information is up to particular Chore
> constructors and by and large we don't get any log messages when things can
> get started, even if the period is something impossibly long (e.g. 3000 days).
> When we go to schedule the chore here:
> {code}
> if (chore.getPeriod() <= 0) {
> LOG.info("The period is {} seconds, {} is disabled",
> chore.getPeriod(), chore.getName());
> return false;
> }
> {code}
> we should add an else clause that says it's enabled. It looks like we could
> then just call chore.toString to get the proper details about the chore and
> its period.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23287) LogCleaner is not added to choreService
[
https://issues.apache.org/jira/browse/HBASE-23287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973588#comment-16973588
]
Sean Busbey commented on HBASE-23287:
-
I'll get this pushed and work through RCs later this week.
> LogCleaner is not added to choreService
> ---
>
> Key: HBASE-23287
> URL: https://issues.apache.org/jira/browse/HBASE-23287
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.6.0, 1.4.11
>Reporter: Baiqiang Zhao
>Assignee: Baiqiang Zhao
>Priority: Blocker
> Attachments: HBASE-23287.branch-1.0001.patch
>
>
> We found /hbase/oldWALs keeps growing, and we never use replication feature.
> Then we found OldWALsCleaner thread is always in WAITTING state :
> {code:java}
> Thread 176 (OldWALsCleaner-1): State: WAITING Blocked count: 0 Waited count:
> 1 Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2ef05d36
> Stack: sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> org.apache.hadoop.hbase.master.cleaner.LogCleaner.deleteFile(LogCleaner.java:185)
>
> org.apache.hadoop.hbase.master.cleaner.LogCleaner.access$000(LogCleaner.java:48)
> org.apache.hadoop.hbase.master.cleaner.LogCleaner$1.run(LogCleaner.java:161)
> java.lang.Thread.run(Thread.java:745)
> {code}
> Finally, we found that LogCleaner is not added to choreService, so LogCleaner
> doesn't run periodically.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23287) LogCleaner is not added to choreService
[
https://issues.apache.org/jira/browse/HBASE-23287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-23287:
Priority: Blocker (was: Critical)
> LogCleaner is not added to choreService
> ---
>
> Key: HBASE-23287
> URL: https://issues.apache.org/jira/browse/HBASE-23287
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.6.0, 1.4.11
>Reporter: Baiqiang Zhao
>Assignee: Baiqiang Zhao
>Priority: Blocker
> Attachments: HBASE-23287.branch-1.0001.patch
>
>
> We found /hbase/oldWALs keeps growing, and we never use replication feature.
> Then we found OldWALsCleaner thread is always in WAITTING state :
> {code:java}
> Thread 176 (OldWALsCleaner-1): State: WAITING Blocked count: 0 Waited count:
> 1 Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2ef05d36
> Stack: sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> org.apache.hadoop.hbase.master.cleaner.LogCleaner.deleteFile(LogCleaner.java:185)
>
> org.apache.hadoop.hbase.master.cleaner.LogCleaner.access$000(LogCleaner.java:48)
> org.apache.hadoop.hbase.master.cleaner.LogCleaner$1.run(LogCleaner.java:161)
> java.lang.Thread.run(Thread.java:745)
> {code}
> Finally, we found that LogCleaner is not added to choreService, so LogCleaner
> doesn't run periodically.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23283) Provide clear and consistent logging about the period of enabled chores
[
https://issues.apache.org/jira/browse/HBASE-23283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973313#comment-16973313
]
Sean Busbey commented on HBASE-23283:
-
Looks great!
+1
> Provide clear and consistent logging about the period of enabled chores
> ---
>
> Key: HBASE-23283
> URL: https://issues.apache.org/jira/browse/HBASE-23283
> Project: HBase
> Issue Type: Improvement
> Components: Operability
>Affects Versions: 3.0.0, 2.3.0, 1.7.0
>Reporter: Sean Busbey
>Assignee: Mingliang Liu
>Priority: Minor
> Attachments: HBASE-23283.000.patch
>
>
> Similar to HBASE-23038, we should always log info about our enabled chores.
> Right now wether or not we get some information is up to particular Chore
> constructors and by and large we don't get any log messages when things can
> get started, even if the period is something impossibly long (e.g. 3000 days).
> When we go to schedule the chore here:
> {code}
> if (chore.getPeriod() <= 0) {
> LOG.info("The period is {} seconds, {} is disabled",
> chore.getPeriod(), chore.getName());
> return false;
> }
> {code}
> we should add an else clause that says it's enabled. It looks like we could
> then just call chore.toString to get the proper details about the chore and
> its period.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23283) Provide clear and consistent logging about the period of enabled chores
[
https://issues.apache.org/jira/browse/HBASE-23283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-23283:
Priority: Minor (was: Major)
> Provide clear and consistent logging about the period of enabled chores
> ---
>
> Key: HBASE-23283
> URL: https://issues.apache.org/jira/browse/HBASE-23283
> Project: HBase
> Issue Type: Improvement
> Components: Operability
>Affects Versions: 3.0.0, 2.3.0, 1.7.0
>Reporter: Sean Busbey
>Priority: Minor
>
> Similar to HBASE-23038, we should always log info about our enabled chores.
> Right now wether or not we get some information is up to particular Chore
> constructors and by and large we don't get any log messages when things can
> get started, even if the period is something impossibly long (e.g. 3000 days).
> When we go to schedule the chore here:
> {code}
> if (chore.getPeriod() <= 0) {
> LOG.info("The period is {} seconds, {} is disabled",
> chore.getPeriod(), chore.getName());
> return false;
> }
> {code}
> we should add an else clause that says it's enabled. It looks like we could
> then just call chore.toString to get the proper details about the chore and
> its period.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-23283) Provide clear and consistent logging about the period of enabled chores
Sean Busbey created HBASE-23283:
---
Summary: Provide clear and consistent logging about the period of
enabled chores
Key: HBASE-23283
URL: https://issues.apache.org/jira/browse/HBASE-23283
Project: HBase
Issue Type: Task
Components: Operability
Affects Versions: 3.0.0, 2.3.0, 1.7.0
Reporter: Sean Busbey
Similar to HBASE-23038, we should always log info about our enabled chores.
Right now wether or not we get some information is up to particular Chore
constructors and by and large we don't get any log messages when things can get
started, even if the period is something impossibly long (e.g. 3000 days).
When we go to schedule the chore here:
{code}
if (chore.getPeriod() <= 0) {
LOG.info("The period is {} seconds, {} is disabled", chore.getPeriod(),
chore.getName());
return false;
}
{code}
we should add an else clause that says it's enabled. It looks like we could
then just call chore.toString to get the proper details about the chore and its
period.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23283) Provide clear and consistent logging about the period of enabled chores
[
https://issues.apache.org/jira/browse/HBASE-23283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-23283:
Issue Type: Improvement (was: Task)
> Provide clear and consistent logging about the period of enabled chores
> ---
>
> Key: HBASE-23283
> URL: https://issues.apache.org/jira/browse/HBASE-23283
> Project: HBase
> Issue Type: Improvement
> Components: Operability
>Affects Versions: 3.0.0, 2.3.0, 1.7.0
>Reporter: Sean Busbey
>Priority: Major
>
> Similar to HBASE-23038, we should always log info about our enabled chores.
> Right now wether or not we get some information is up to particular Chore
> constructors and by and large we don't get any log messages when things can
> get started, even if the period is something impossibly long (e.g. 3000 days).
> When we go to schedule the chore here:
> {code}
> if (chore.getPeriod() <= 0) {
> LOG.info("The period is {} seconds, {} is disabled",
> chore.getPeriod(), chore.getName());
> return false;
> }
> {code}
> we should add an else clause that says it's enabled. It looks like we could
> then just call chore.toString to get the proper details about the chore and
> its period.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23069) periodic dependency bump for Sep 2019
[ https://issues.apache.org/jira/browse/HBASE-23069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-23069: Fix Version/s: (was: 1.7.0) 1.6.0 > periodic dependency bump for Sep 2019 > - > > Key: HBASE-23069 > URL: https://issues.apache.org/jira/browse/HBASE-23069 > Project: HBase > Issue Type: Improvement > Components: dependencies, hbase-thirdparty >Reporter: Sean Busbey >Priority: Critical > Fix For: 3.0.0, 2.3.0, 1.6.0 > > > we should do a pass to see if there are any dependencies we can bump. (also > follow-on we should automate this check) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23220) Release 1.6.0
[ https://issues.apache.org/jira/browse/HBASE-23220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971742#comment-16971742 ] Sean Busbey commented on HBASE-23220: - based on this thread: https://lists.apache.org/thread.html/cc66ce601a0bcd4fc809139f3f79f63607ee9f5bb4dcffcb1a92b2e6@%3Cdev.hbase.apache.org%3E I'm going to repurpose this for relesing 1.6.0. If anyone wants a 1.5.1 please feel free, but start a new jira. > Release 1.6.0 > - > > Key: HBASE-23220 > URL: https://issues.apache.org/jira/browse/HBASE-23220 > Project: HBase > Issue Type: Task > Components: community >Affects Versions: 1.5.1 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 1.6.0 > > > let's roll 1.6.0 to get HBASE-23174 out on recent branch-1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23220) Release 1.5.1
[ https://issues.apache.org/jira/browse/HBASE-23220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-23220: Description: let's roll 1.6.0 to get HBASE-23174 out on recent branch-1 (was: let's roll 1.5.1 to get HBASE-23174 out on 1.5.z.) > Release 1.5.1 > - > > Key: HBASE-23220 > URL: https://issues.apache.org/jira/browse/HBASE-23220 > Project: HBase > Issue Type: Task > Components: community >Affects Versions: 1.5.1 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 1.5.1 > > > let's roll 1.6.0 to get HBASE-23174 out on recent branch-1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23220) Release 1.5.1
[ https://issues.apache.org/jira/browse/HBASE-23220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-23220: Fix Version/s: (was: 1.5.1) 1.6.0 > Release 1.5.1 > - > > Key: HBASE-23220 > URL: https://issues.apache.org/jira/browse/HBASE-23220 > Project: HBase > Issue Type: Task > Components: community >Affects Versions: 1.5.1 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 1.6.0 > > > let's roll 1.6.0 to get HBASE-23174 out on recent branch-1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23220) Release 1.6.0
[ https://issues.apache.org/jira/browse/HBASE-23220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-23220: Summary: Release 1.6.0 (was: Release 1.5.1) > Release 1.6.0 > - > > Key: HBASE-23220 > URL: https://issues.apache.org/jira/browse/HBASE-23220 > Project: HBase > Issue Type: Task > Components: community >Affects Versions: 1.5.1 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 1.6.0 > > > let's roll 1.6.0 to get HBASE-23174 out on recent branch-1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23274) Remove auto generated files by thrift from project
[ https://issues.apache.org/jira/browse/HBASE-23274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16970955#comment-16970955 ] Sean Busbey commented on HBASE-23274: - I think we add them for now to avoid needing a thrift compiler at build time. Or are we using a maven plugin now? > Remove auto generated files by thrift from project > -- > > Key: HBASE-23274 > URL: https://issues.apache.org/jira/browse/HBASE-23274 > Project: HBase > Issue Type: Bug >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > > Now we have added auto generated files by thrift in the project, and we will > regenerate these files whenever we add interfaces. It might be better to > generate these files automatically at compile time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-15611) add examples to shell docs
[
https://issues.apache.org/jira/browse/HBASE-15611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16970384#comment-16970384
]
Sean Busbey edited comment on HBASE-15611 at 11/8/19 4:47 PM:
--
Eventually I'll put this into the MOB section for troubleshooting.
"scan for mob references rather than for the values themselves"
{code}
hbase(main):074:0> scan 'example_table', {LIMIT => 1, CACHE_BLOCKS => false,
ATTRIBUTES => { 'hbase.mob.scan.raw' => '1', 'hbase.mob.scan.ref.only' => '1' }
}
{code}
"count the number of rows with mob references and the number of mob reference
cells"
{code}
hbase(main):031:0> row_count = 0
=> 0
hbase(main):032:0> cell_count = 0
=> 0
hbase(main):033:0> table = get_table('clone_table')
0 row(s) in 0. seconds
=> Hbase::Table - clone_table
hbase(main):034:0> rows = table._get_scanner({CACHE_BLOCKS => false, ATTRIBUTES
=> { 'hbase.mob.scan.raw' => '1', 'hbase.mob.scan.ref.only' => '1' } }).iterator
=> #<#:0x7da887f>
hbase(main):035:0> rows.each do | row | row_count+=1; row.list.each do | cell |
cell_count+=1; end end
hbase(main):036:0> p row_count
1
hbase(main):037:0> p cell_count
52360
{code}
Very important warning: if you use this without at least one column with MOB
enabled, you will get a full table scan with no filtering.
was (Author: busbey):
Eventually I'll put this into the MOB section for troubleshooting.
"scan for mob references rather than for the values themselves"
{code}
hbase(main):074:0> scan 'example_table', {LIMIT => 1, CACHE_BLOCKS => false,
ATTRIBUTES => { 'hbase.mob.scan.raw' => '1', 'hbase.mob.scan.ref.only' => '1' }
}
{code}
"count the number of rows with mob references and the number of mob reference
cells"
{code}
hbase(main):031:0> row_count = 0
=> 0
hbase(main):032:0> cell_count = 0
=> 0
hbase(main):033:0> table = get_table('clone_table')
0 row(s) in 0. seconds
=> Hbase::Table - clone_table
hbase(main):034:0> rows = table._get_scanner({CACHE_BLOCKS => false, ATTRIBUTES
=> { 'hbase.mob.scan.raw' => '1', 'hbase.mob.scan.ref.only' => '1' } }).iterator
=> #<#:0x7da887f>
hbase(main):035:0> rows.each do | row | row_count+=1; row.list.each do | cell |
cell_count+=1; end end
hbase(main):036:0> p row_count
1
hbase(main):037:0> p cell_count
52360
{code}
> add examples to shell docs
> ---
>
> Key: HBASE-15611
> URL: https://issues.apache.org/jira/browse/HBASE-15611
> Project: HBase
> Issue Type: Improvement
> Components: documentation, shell
>Reporter: Sean Busbey
>Priority: Major
> Labels: beginner
>
> It would be nice if our shell documentation included some additional examples
> of operational tasks one can perform.
> things to include to come in comments. when we have a patch to submit we can
> update the jira summary to better reflect what scope we end up with.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-15611) add examples to shell docs
[
https://issues.apache.org/jira/browse/HBASE-15611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16970384#comment-16970384
]
Sean Busbey commented on HBASE-15611:
-
Eventually I'll put this into the MOB section for troubleshooting.
"scan for mob references rather than for the values themselves"
{code}
hbase(main):074:0> scan 'example_table', {LIMIT => 1, CACHE_BLOCKS => false,
ATTRIBUTES => { 'hbase.mob.scan.raw' => '1', 'hbase.mob.scan.ref.only' => '1' }
}
{code}
"count the number of rows with mob references and the number of mob reference
cells"
{code}
hbase(main):031:0> row_count = 0
=> 0
hbase(main):032:0> cell_count = 0
=> 0
hbase(main):033:0> table = get_table('clone_table')
0 row(s) in 0. seconds
=> Hbase::Table - clone_table
hbase(main):034:0> rows = table._get_scanner({CACHE_BLOCKS => false, ATTRIBUTES
=> { 'hbase.mob.scan.raw' => '1', 'hbase.mob.scan.ref.only' => '1' } }).iterator
=> #<#:0x7da887f>
hbase(main):035:0> rows.each do | row | row_count+=1; row.list.each do | cell |
cell_count+=1; end end
hbase(main):036:0> p row_count
1
hbase(main):037:0> p cell_count
52360
{code}
> add examples to shell docs
> ---
>
> Key: HBASE-15611
> URL: https://issues.apache.org/jira/browse/HBASE-15611
> Project: HBase
> Issue Type: Improvement
> Components: documentation, shell
>Reporter: Sean Busbey
>Priority: Major
> Labels: beginner
>
> It would be nice if our shell documentation included some additional examples
> of operational tasks one can perform.
> things to include to come in comments. when we have a patch to submit we can
> update the jira summary to better reflect what scope we end up with.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-23228) Allow for jdk8 specific modules on branch-1 in precommit/nightly testing
[ https://issues.apache.org/jira/browse/HBASE-23228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-23228. - Fix Version/s: 2.2.3 2.1.8 1.4.12 1.6.0 2.3.0 3.0.0 Resolution: Fixed > Allow for jdk8 specific modules on branch-1 in precommit/nightly testing > > > Key: HBASE-23228 > URL: https://issues.apache.org/jira/browse/HBASE-23228 > Project: HBase > Issue Type: Improvement > Components: build, test >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 3.0.0, 2.3.0, 1.6.0, 1.4.12, 2.1.8, 2.2.3 > > > At least 1 feature backport is waiting on proper handling of jdk8 activated > modules for our yetus personality (HBASE-22114 tinylfu). > Implement the general handling here so that we don't have to worry about > pushes to the PR branch overwriting it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23248) hbase openjdk11 compile error
[ https://issues.apache.org/jira/browse/HBASE-23248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969804#comment-16969804 ] Sean Busbey commented on HBASE-23248: - bq. [WARNING] Some problems were encountered while building the effective model for org.apache.hbase:hbase-build-configuration:pom:2.1.0-cdh6.3.0 you're building with a vendor's fork of the project. AFAIK you can't build it with jdk11. You should use the vendor's support mechanisms if you need to get compiling with jdk11 to work for some reason. The first release from the apache hbase project that is expected to build with jdk11 will be the upcoming Apache HBase 2.3.0. > hbase openjdk11 compile error > -- > > Key: HBASE-23248 > URL: https://issues.apache.org/jira/browse/HBASE-23248 > Project: HBase > Issue Type: Bug > Components: build, java >Reporter: jackylau >Priority: Major > Attachments: log > > > i find this > [https://stackoverflow.com/questions/49398894/unable-to-compile-simple-java-10-java-11-project-with-maven/55047110#55047110], > but it still can not solve this problem > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-shade-plugin:3.0.0:shade (default) on project > hbase-protocol-shaded: Error creating shaded jar: null: > IllegalArgumentException -> [Help 1][ERROR] Failed to execute goal > org.apache.maven.plugins:maven-shade-plugin:3.0.0:shade (default) on project > hbase-protocol-shaded: Error creating shaded jar: null: > IllegalArgumentException -> [Help > 1]org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute > goal org.apache.maven.plugins:maven-shade-plugin:3.0.0:shade (default) on > project hbase-protocol-shaded: Error creating shaded jar: null at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116) > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80) > at > org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) > at > org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128) > at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307) at > org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193) at > org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106) at > org.apache.maven.cli.MavenCli.execute(MavenCli.java:863) at > org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288) at > org.apache.maven.cli.MavenCli.main(MavenCli.java:199) at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) at > org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) > at > org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) > at > org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) > at > org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)Caused > by: org.apache.maven.plugin.MojoExecutionException: Error creating shaded > jar: null at > org.apache.maven.plugins.shade.mojo.ShadeMojo.execute(ShadeMojo.java:546) at > org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207) > ... 20 moreCaused by: java.lang.IllegalArgumentException at > org.objectweb.asm.ClassReader.(Unknown Source) at > org.objectweb.asm.ClassReader.(Unknown Source) at > org.objectweb.asm.ClassReader.(Unknown Source) at > org.vafer.jdependency.Clazzpath.addClazzpathUnit(Clazzpath.java:201) at > org.vafer.jdependency.Clazzpath.addClazzpathUnit(Clazzpath.java:132) at > org.apache.maven.plugins.shade.filter.MinijarFilter.(MinijarFilter.java:95) > at > org.apache.maven.plugins.shade.mojo.ShadeMojo.getFilters(ShadeMojo.java:826) > at org.apache.maven.plugins.shade.mojo.ShadeMojo.execute(ShadeMojo.java:434) > ... 22 more -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23185) High cpu usage because getTable()#put() gets config value every time
[ https://issues.apache.org/jira/browse/HBASE-23185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969662#comment-16969662 ] Sean Busbey commented on HBASE-23185: - Yeah all passes https://builds.apache.org/job/HBase-Flaky-Tests/job/branch-1.3/test_results_analyzer/ > High cpu usage because getTable()#put() gets config value every time > > > Key: HBASE-23185 > URL: https://issues.apache.org/jira/browse/HBASE-23185 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 1.5.0, 1.4.10, 1.2.12, 1.3.5 >Reporter: Shinya Yoshida >Assignee: Shinya Yoshida >Priority: Major > Labels: performance > Fix For: 1.6.0 > > Attachments: Screenshot from 2019-10-18 12-38-14.png, Screenshot from > 2019-10-18 13-03-24.png > > > When we analyzed the performance of our hbase application with many puts, we > found that Configuration methods use many CPU resources: > !Screenshot from 2019-10-18 12-38-14.png|width=460,height=205! > As you can see, getTable().put() is calling Configuration methods which cause > regex or synchronization by Hashtable. > This should not happen in 0.99.2 because > https://issues.apache.org/jira/browse/HBASE-12128 addressed such an issue. > However, it's reproducing nowadays by bugs or leakages after many code > evoluations between 0.9x and 1.x. > # > [https://github.com/apache/hbase/blob/dd9eadb00f9dcd071a246482a11dfc7d63845f00/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java#L369-L374] > ** finishSetup is called every new HTable() e.g. every con.getTable() > ** So getInt is called everytime and it does regex > # > [https://github.com/apache/hbase/blob/dd9eadb00f9dcd071a246482a11dfc7d63845f00/hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutatorImpl.java#L115] > ** BufferedMutatorImpl is created every first put for HTable e.g. > con.getTable().put() > ** Create ConnectionConf every time in BufferedMutatorImpl constructor > ** ConnectionConf gets config value in the constructor > # > [https://github.com/apache/hbase/blob/dd9eadb00f9dcd071a246482a11dfc7d63845f00/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java#L326] > ** AsyncProcess is created in BufferedMutatorImpl constructor, so new > AsyncProcess is created by con.getTable().put() > ** AsyncProcess parse many configurations > So, con.getTable().put() is heavy operation for CPU because of getting config > value. > > With in-house patch for this issue, we observed about 10% improvement on > max-throughput (e.g. CPU usage) at client-side: > !Screenshot from 2019-10-18 13-03-24.png|width=508,height=223! > > Seems branch-2 is not affected because client implementation has been changed > dramatically. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23185) High cpu usage because getTable()#put() gets config value every time
[ https://issues.apache.org/jira/browse/HBASE-23185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969661#comment-16969661 ] Sean Busbey commented on HBASE-23185: - If it's not in that list at all it probably passed every time. Check the Jenkins test summary for the job to confirm > High cpu usage because getTable()#put() gets config value every time > > > Key: HBASE-23185 > URL: https://issues.apache.org/jira/browse/HBASE-23185 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 1.5.0, 1.4.10, 1.2.12, 1.3.5 >Reporter: Shinya Yoshida >Assignee: Shinya Yoshida >Priority: Major > Labels: performance > Fix For: 1.6.0 > > Attachments: Screenshot from 2019-10-18 12-38-14.png, Screenshot from > 2019-10-18 13-03-24.png > > > When we analyzed the performance of our hbase application with many puts, we > found that Configuration methods use many CPU resources: > !Screenshot from 2019-10-18 12-38-14.png|width=460,height=205! > As you can see, getTable().put() is calling Configuration methods which cause > regex or synchronization by Hashtable. > This should not happen in 0.99.2 because > https://issues.apache.org/jira/browse/HBASE-12128 addressed such an issue. > However, it's reproducing nowadays by bugs or leakages after many code > evoluations between 0.9x and 1.x. > # > [https://github.com/apache/hbase/blob/dd9eadb00f9dcd071a246482a11dfc7d63845f00/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java#L369-L374] > ** finishSetup is called every new HTable() e.g. every con.getTable() > ** So getInt is called everytime and it does regex > # > [https://github.com/apache/hbase/blob/dd9eadb00f9dcd071a246482a11dfc7d63845f00/hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutatorImpl.java#L115] > ** BufferedMutatorImpl is created every first put for HTable e.g. > con.getTable().put() > ** Create ConnectionConf every time in BufferedMutatorImpl constructor > ** ConnectionConf gets config value in the constructor > # > [https://github.com/apache/hbase/blob/dd9eadb00f9dcd071a246482a11dfc7d63845f00/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java#L326] > ** AsyncProcess is created in BufferedMutatorImpl constructor, so new > AsyncProcess is created by con.getTable().put() > ** AsyncProcess parse many configurations > So, con.getTable().put() is heavy operation for CPU because of getting config > value. > > With in-house patch for this issue, we observed about 10% improvement on > max-throughput (e.g. CPU usage) at client-side: > !Screenshot from 2019-10-18 13-03-24.png|width=508,height=223! > > Seems branch-2 is not affected because client implementation has been changed > dramatically. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HBASE-23228) Allow for jdk8 specific modules on branch-1 in precommit/nightly testing
[ https://issues.apache.org/jira/browse/HBASE-23228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-23228 started by Sean Busbey. --- > Allow for jdk8 specific modules on branch-1 in precommit/nightly testing > > > Key: HBASE-23228 > URL: https://issues.apache.org/jira/browse/HBASE-23228 > Project: HBase > Issue Type: Improvement > Components: build, test >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > > At least 1 feature backport is waiting on proper handling of jdk8 activated > modules for our yetus personality (HBASE-22114 tinylfu). > Implement the general handling here so that we don't have to worry about > pushes to the PR branch overwriting it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23248) hbase openjdk11 compile error
[
https://issues.apache.org/jira/browse/HBASE-23248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968693#comment-16968693
]
Sean Busbey commented on HBASE-23248:
-
Can you get some more specifics about your environment? this morning I was able
to build master ref f0f7fae400
{code}
hbase busbey$ mvn -version && mvn -DskipTests package assembly:single
Apache Maven 3.6.0 (97c98ec64a1fdfee7767ce5ffb20918da4f719f3;
2018-10-24T13:41:47-05:00)
Maven home: /usr/local/Cellar/maven/3.6.0/libexec
Java version: 11.0.2, vendor: Oracle Corporation, runtime:
/Library/Java/JavaVirtualMachines/openjdk-11.0.2.jdk/Contents/Home
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac"
[INFO] Scanning for projects...
[INFO]
[INFO] Detecting the operating system and CPU architecture
[INFO]
[INFO] os.detected.name: osx
[INFO] os.detected.arch: x86_64
[INFO] os.detected.version: 10.13
[INFO] os.detected.version.major: 10
[INFO] os.detected.version.minor: 13
[INFO] os.detected.classifier: osx-x86_64
[INFO]
[INFO] Reactor Build Order:
[INFO]
...
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 09:21 min
[INFO] Finished at: 2019-11-06T02:54:09-06:00
[INFO]
{code}
> hbase openjdk11 compile error
> --
>
> Key: HBASE-23248
> URL: https://issues.apache.org/jira/browse/HBASE-23248
> Project: HBase
> Issue Type: Bug
> Components: build, java
>Reporter: jackylau
>Priority: Major
>
> i find this
> [https://stackoverflow.com/questions/49398894/unable-to-compile-simple-java-10-java-11-project-with-maven/55047110#55047110],
> but it still can not solve this problem
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.0.0:shade (default) on project
> hbase-protocol-shaded: Error creating shaded jar: null:
> IllegalArgumentException -> [Help 1][ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.0.0:shade (default) on project
> hbase-protocol-shaded: Error creating shaded jar: null:
> IllegalArgumentException -> [Help
> 1]org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.apache.maven.plugins:maven-shade-plugin:3.0.0:shade (default) on
> project hbase-protocol-shaded: Error creating shaded jar: null at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
> at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
> at
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307) at
> org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193) at
> org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106) at
> org.apache.maven.cli.MavenCli.execute(MavenCli.java:863) at
> org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288) at
> org.apache.maven.cli.MavenCli.main(MavenCli.java:199) at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method) at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566) at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)Caused
> by: org.apache.maven.plugin.MojoExecutionException: Error creating shaded
> jar: null at
> org.apache.maven.plugins.shade.mojo.ShadeMojo.execute(ShadeMojo.java:546) at
>
[jira] [Resolved] (HBASE-21458) Error: Could not find or load main class org.apache.hadoop.hbase.util.GetJavaProperty
[ https://issues.apache.org/jira/browse/HBASE-21458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-21458. - Fix Version/s: 2.2.3 2.1.8 2.3.0 3.0.0 Resolution: Fixed > Error: Could not find or load main class > org.apache.hadoop.hbase.util.GetJavaProperty > - > > Key: HBASE-21458 > URL: https://issues.apache.org/jira/browse/HBASE-21458 > Project: HBase > Issue Type: Bug >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3 > > Attachments: HBASE-21458.branch-2.1.001.patch, > HBASE-21458.branch-2.1.002.patch > > > I get this when I run bin/hbase classpath whether a built checkout or an > undone tarball. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-21458) Error: Could not find or load main class org.apache.hadoop.hbase.util.GetJavaProperty
[ https://issues.apache.org/jira/browse/HBASE-21458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-21458: Component/s: Client build > Error: Could not find or load main class > org.apache.hadoop.hbase.util.GetJavaProperty > - > > Key: HBASE-21458 > URL: https://issues.apache.org/jira/browse/HBASE-21458 > Project: HBase > Issue Type: Bug > Components: build, Client >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3 > > Attachments: HBASE-21458.branch-2.1.001.patch, > HBASE-21458.branch-2.1.002.patch > > > I get this when I run bin/hbase classpath whether a built checkout or an > undone tarball. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-21458) Error: Could not find or load main class org.apache.hadoop.hbase.util.GetJavaProperty
[ https://issues.apache.org/jira/browse/HBASE-21458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-21458: Priority: Minor (was: Major) > Error: Could not find or load main class > org.apache.hadoop.hbase.util.GetJavaProperty > - > > Key: HBASE-21458 > URL: https://issues.apache.org/jira/browse/HBASE-21458 > Project: HBase > Issue Type: Bug > Components: build, Client >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Minor > Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3 > > Attachments: HBASE-21458.branch-2.1.001.patch, > HBASE-21458.branch-2.1.002.patch > > > I get this when I run bin/hbase classpath whether a built checkout or an > undone tarball. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23248) hbase openjdk11 compile error
[ https://issues.apache.org/jira/browse/HBASE-23248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-23248: Component/s: java build > hbase openjdk11 compile error > -- > > Key: HBASE-23248 > URL: https://issues.apache.org/jira/browse/HBASE-23248 > Project: HBase > Issue Type: Bug > Components: build, java >Reporter: jackylau >Priority: Major > > i find this > [https://stackoverflow.com/questions/49398894/unable-to-compile-simple-java-10-java-11-project-with-maven/55047110#55047110], > but it still can not solve this problem > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-shade-plugin:3.0.0:shade (default) on project > hbase-protocol-shaded: Error creating shaded jar: null: > IllegalArgumentException -> [Help 1][ERROR] Failed to execute goal > org.apache.maven.plugins:maven-shade-plugin:3.0.0:shade (default) on project > hbase-protocol-shaded: Error creating shaded jar: null: > IllegalArgumentException -> [Help > 1]org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute > goal org.apache.maven.plugins:maven-shade-plugin:3.0.0:shade (default) on > project hbase-protocol-shaded: Error creating shaded jar: null at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116) > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80) > at > org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) > at > org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128) > at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307) at > org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193) at > org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106) at > org.apache.maven.cli.MavenCli.execute(MavenCli.java:863) at > org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288) at > org.apache.maven.cli.MavenCli.main(MavenCli.java:199) at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) at > org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) > at > org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) > at > org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) > at > org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)Caused > by: org.apache.maven.plugin.MojoExecutionException: Error creating shaded > jar: null at > org.apache.maven.plugins.shade.mojo.ShadeMojo.execute(ShadeMojo.java:546) at > org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207) > ... 20 moreCaused by: java.lang.IllegalArgumentException at > org.objectweb.asm.ClassReader.(Unknown Source) at > org.objectweb.asm.ClassReader.(Unknown Source) at > org.objectweb.asm.ClassReader.(Unknown Source) at > org.vafer.jdependency.Clazzpath.addClazzpathUnit(Clazzpath.java:201) at > org.vafer.jdependency.Clazzpath.addClazzpathUnit(Clazzpath.java:132) at > org.apache.maven.plugins.shade.filter.MinijarFilter.(MinijarFilter.java:95) > at > org.apache.maven.plugins.shade.mojo.ShadeMojo.getFilters(ShadeMojo.java:826) > at org.apache.maven.plugins.shade.mojo.ShadeMojo.execute(ShadeMojo.java:434) > ... 22 more -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23248) hbase openjdk11 compile error
[
https://issues.apache.org/jira/browse/HBASE-23248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967972#comment-16967972
]
Sean Busbey commented on HBASE-23248:
-
what's the full output of {{mvn --version}}
> hbase openjdk11 compile error
> --
>
> Key: HBASE-23248
> URL: https://issues.apache.org/jira/browse/HBASE-23248
> Project: HBase
> Issue Type: Bug
>Reporter: jackylau
>Priority: Major
>
> i find this
> [https://stackoverflow.com/questions/49398894/unable-to-compile-simple-java-10-java-11-project-with-maven/55047110#55047110],
> but it still can not solve this problem
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.0.0:shade (default) on project
> hbase-protocol-shaded: Error creating shaded jar: null:
> IllegalArgumentException -> [Help 1][ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.0.0:shade (default) on project
> hbase-protocol-shaded: Error creating shaded jar: null:
> IllegalArgumentException -> [Help
> 1]org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.apache.maven.plugins:maven-shade-plugin:3.0.0:shade (default) on
> project hbase-protocol-shaded: Error creating shaded jar: null at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
> at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
> at
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307) at
> org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193) at
> org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106) at
> org.apache.maven.cli.MavenCli.execute(MavenCli.java:863) at
> org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288) at
> org.apache.maven.cli.MavenCli.main(MavenCli.java:199) at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method) at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566) at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)Caused
> by: org.apache.maven.plugin.MojoExecutionException: Error creating shaded
> jar: null at
> org.apache.maven.plugins.shade.mojo.ShadeMojo.execute(ShadeMojo.java:546) at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
> ... 20 moreCaused by: java.lang.IllegalArgumentException at
> org.objectweb.asm.ClassReader.(Unknown Source) at
> org.objectweb.asm.ClassReader.(Unknown Source) at
> org.objectweb.asm.ClassReader.(Unknown Source) at
> org.vafer.jdependency.Clazzpath.addClazzpathUnit(Clazzpath.java:201) at
> org.vafer.jdependency.Clazzpath.addClazzpathUnit(Clazzpath.java:132) at
> org.apache.maven.plugins.shade.filter.MinijarFilter.(MinijarFilter.java:95)
> at
> org.apache.maven.plugins.shade.mojo.ShadeMojo.getFilters(ShadeMojo.java:826)
> at org.apache.maven.plugins.shade.mojo.ShadeMojo.execute(ShadeMojo.java:434)
> ... 22 more
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-23250) Log message about CleanerChore delegate initialization should be at INFO
Sean Busbey created HBASE-23250:
---
Summary: Log message about CleanerChore delegate initialization
should be at INFO
Key: HBASE-23250
URL: https://issues.apache.org/jira/browse/HBASE-23250
Project: HBase
Issue Type: Task
Components: master, Operability
Reporter: Sean Busbey
Right now we log a DEBUG message as we initialize the delegates that decide on
cleaner decisions:
{code}
2019-10-30 07:30:01,594 DEBUG
org.apache.hadoop.hbase.master.cleaner.CleanerChore: initialize
cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner
2019-10-30 07:30:01,597 DEBUG
org.apache.hadoop.hbase.master.cleaner.CleanerChore: initialize
cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotLogCleaner
2019-10-30 07:30:01,600 DEBUG
org.apache.hadoop.hbase.master.cleaner.CleanerChore: initialize
cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner
2019-10-30 07:30:01,601 DEBUG
org.apache.hadoop.hbase.master.cleaner.CleanerChore: initialize
cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner
2019-10-30 07:30:01,602 DEBUG
org.apache.hadoop.hbase.master.cleaner.CleanerChore: initialize
cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner
{code}
Done here in CleanerChore.java:
{code}
private void initCleanerChain(String confKey) {
this.cleanersChain = new LinkedList<>();
String[] logCleaners = conf.getStrings(confKey);
if (logCleaners != null) {
for (String className : logCleaners) {
T logCleaner = newFileCleaner(className, conf);
if (logCleaner != null) {
LOG.debug("Initialize cleaner={}", className);
this.cleanersChain.add(logCleaner);
}
}
}
}
{code}
This message should be at INFO. it doesn't happen often and knowing what
plugins decide about file deletions is important all the time.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23250) Log message about CleanerChore delegate initialization should be at INFO
[
https://issues.apache.org/jira/browse/HBASE-23250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-23250:
Labels: beginner (was: )
> Log message about CleanerChore delegate initialization should be at INFO
>
>
> Key: HBASE-23250
> URL: https://issues.apache.org/jira/browse/HBASE-23250
> Project: HBase
> Issue Type: Task
> Components: master, Operability
>Reporter: Sean Busbey
>Priority: Minor
> Labels: beginner
>
> Right now we log a DEBUG message as we initialize the delegates that decide
> on cleaner decisions:
> {code}
> 2019-10-30 07:30:01,594 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: initialize
> cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner
> 2019-10-30 07:30:01,597 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: initialize
> cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotLogCleaner
> 2019-10-30 07:30:01,600 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: initialize
> cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner
> 2019-10-30 07:30:01,601 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: initialize
> cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner
> 2019-10-30 07:30:01,602 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: initialize
> cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner
> {code}
> Done here in CleanerChore.java:
> {code}
> private void initCleanerChain(String confKey) {
> this.cleanersChain = new LinkedList<>();
> String[] logCleaners = conf.getStrings(confKey);
> if (logCleaners != null) {
> for (String className : logCleaners) {
> T logCleaner = newFileCleaner(className, conf);
> if (logCleaner != null) {
> LOG.debug("Initialize cleaner={}", className);
> this.cleanersChain.add(logCleaner);
> }
> }
> }
> }
> {code}
> This message should be at INFO. it doesn't happen often and knowing what
> plugins decide about file deletions is important all the time.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23210) Backport HBASE-15519 (Add per-user metrics) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-23210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16965443#comment-16965443 ] Sean Busbey commented on HBASE-23210: - Okay cool. Thanks. > Backport HBASE-15519 (Add per-user metrics) to branch-1 > --- > > Key: HBASE-23210 > URL: https://issues.apache.org/jira/browse/HBASE-23210 > Project: HBase > Issue Type: New Feature >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 1.5.1 > > > We will need HBASE-15519 in branch-1 for eventual backport of HBASE-23065. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23210) Backport HBASE-15519 (Add per-user metrics) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-23210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16965440#comment-16965440 ] Sean Busbey commented on HBASE-23210: - Ftr we still do jdk8 builds in nightly (in addition to jdk7) so that we can check findbugs and error prone. If we're going to ignore error prone on branch-1 I will happily remove it. > Backport HBASE-15519 (Add per-user metrics) to branch-1 > --- > > Key: HBASE-23210 > URL: https://issues.apache.org/jira/browse/HBASE-23210 > Project: HBase > Issue Type: New Feature >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 1.5.1 > > > We will need HBASE-15519 in branch-1 for eventual backport of HBASE-23065. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-23244) NPEs running Canary
[
https://issues.apache.org/jira/browse/HBASE-23244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey reopened HBASE-23244:
-
This caused a findbugs failure.
>From precommit report, same thing pinging nightly and all new precommits as a
>branch problem:
{code}
FindBugsmodule:hbase-server
Nullcheck of CanaryTool$RegionTask.region at line 495 of value previously
dereferenced in org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.read() At
CanaryTool.java:495 of value previously dereferenced in
org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.read() At
CanaryTool.java:[line 438]
{code}
> NPEs running Canary
> ---
>
> Key: HBASE-23244
> URL: https://issues.apache.org/jira/browse/HBASE-23244
> Project: HBase
> Issue Type: Bug
> Components: canary
>Affects Versions: 2.2.2
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3
>
>
> There's a few. Here is one:
> {code}
> 19/11/01 23:54:16 ERROR tool.Canary: Run regionMonitor failed
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.tool.CanaryTool.sniff(CanaryTool.java:1582)
> at org.apache.hadoop.hbase.tool.CanaryTool.sniff(CanaryTool.java:1562)
> at
> org.apache.hadoop.hbase.tool.CanaryTool.access$100(CanaryTool.java:124)
> at
> org.apache.hadoop.hbase.tool.CanaryTool$RegionMonitor.run(CanaryTool.java:1366)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (HBASE-23210) Backport HBASE-15519 (Add per-user metrics) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-23210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey reopened HBASE-23210:
-
This change is causing Error Prone to fail branch-1 nightlies:
{code}
[INFO] -
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR]
/home/jenkins/jenkins-slave/workspace/HBase_Nightly_branch-1/component/hbase-hadoop2-compat/src/test/java/org/apache/hadoop/hbase/regionserver/TestMetricsUserSourceImpl.java:[50,29]
[SelfComparison] An object is compared to itself
(see http://errorprone.info/bugpattern/SelfComparison)
{code}
> Backport HBASE-15519 (Add per-user metrics) to branch-1
> ---
>
> Key: HBASE-23210
> URL: https://issues.apache.org/jira/browse/HBASE-23210
> Project: HBase
> Issue Type: New Feature
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 1.5.1
>
>
> We will need HBASE-15519 in branch-1 for eventual backport of HBASE-23065.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23243) [pv2] Filter out SUCCESS procedures; on decent-sized cluster, plethora overwhelms problems
[ https://issues.apache.org/jira/browse/HBASE-23243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16965160#comment-16965160 ] Sean Busbey commented on HBASE-23243: - very interested in tooling related here. it's been on my list for ages but keeps falling too far behind the top. even useful on 1.z > [pv2] Filter out SUCCESS procedures; on decent-sized cluster, plethora > overwhelms problems > -- > > Key: HBASE-23243 > URL: https://issues.apache.org/jira/browse/HBASE-23243 > Project: HBase > Issue Type: Bug > Components: proc-v2, UI >Reporter: Michael Stack >Priority: Major > > Trying to figure some problem procedures using nice UI tools. Listing of > Procedures and Locks shows SUCCESS Procedures and WAITING/RUNNABLEs all in > the same list. Can't find why a SCP is stuck when it has hundreds of > subprocedures all of which have SUCCEEDED but one especially when list > includes other concurrent SCPs > For now will just filter out SUCCESS items. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HBASE-23239) Reporting on status of backing MOB files from client-facing cells
[ https://issues.apache.org/jira/browse/HBASE-23239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-23239 started by Sean Busbey. --- > Reporting on status of backing MOB files from client-facing cells > - > > Key: HBASE-23239 > URL: https://issues.apache.org/jira/browse/HBASE-23239 > Project: HBase > Issue Type: Improvement > Components: mapreduce, mob, Operability >Affects Versions: 2.1.0, 2.0.0, 2.2.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > > We should be able to do some correctness testing of the mob files referenced > by current cells in a mob enabled table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23239) Reporting on status of backing MOB files from client-facing cells
Sean Busbey created HBASE-23239: --- Summary: Reporting on status of backing MOB files from client-facing cells Key: HBASE-23239 URL: https://issues.apache.org/jira/browse/HBASE-23239 Project: HBase Issue Type: Improvement Components: mapreduce, mob, Operability Affects Versions: 2.2.0, 2.0.0, 2.1.0 Reporter: Sean Busbey Assignee: Sean Busbey We should be able to do some correctness testing of the mob files referenced by current cells in a mob enabled table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23229) Update branch-1 to 1.6.0-SNAPSHOT
[ https://issues.apache.org/jira/browse/HBASE-23229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-23229. - Fix Version/s: 1.6.0 Resolution: Fixed > Update branch-1 to 1.6.0-SNAPSHOT > - > > Key: HBASE-23229 > URL: https://issues.apache.org/jira/browse/HBASE-23229 > Project: HBase > Issue Type: Task > Components: build, community >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 1.6.0 > > > since 1.5.0 was released from branch-1 we've had new feature changes land. > update the branch so the next release from it will be 1.6.0. > subsequent 1.5.z releases can be made by creating a new branch from rel/1.5.0 > and cherry-picking desired changes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23233) update CellCounter so that granularity of counts is configurable
Sean Busbey created HBASE-23233: --- Summary: update CellCounter so that granularity of counts is configurable Key: HBASE-23233 URL: https://issues.apache.org/jira/browse/HBASE-23233 Project: HBase Issue Type: Improvement Components: mapreduce, Operability, tooling Reporter: Sean Busbey I don't think I've seen a non-trivial deployment where CellCounter can run without having to change the YARN/MR counter limits. It's always a pain. We should make it so that the granularity of things CellCounter aggregates is configurable and make most of them opt-in so that we can stay under the default limit of counters when folks try the tool out. like < 100 of them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23171) Ensure region state error messages include server making report (branch-2.1)
[
https://issues.apache.org/jira/browse/HBASE-23171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-23171:
Priority: Minor (was: Major)
> Ensure region state error messages include server making report (branch-2.1)
>
>
> Key: HBASE-23171
> URL: https://issues.apache.org/jira/browse/HBASE-23171
> Project: HBase
> Issue Type: Bug
> Components: amv2, Operability
>Affects Versions: 2.1.3
>Reporter: Michael Stack
>Assignee: Sean Busbey
>Priority: Minor
> Fix For: 2.1.8
>
>
> This message is near useless. A RS is carrying a region that is not known to
> master but we don't know which server has it because message doesn't say who
> is complaining (this is fixed in 2.2.x).
> {code}
> 2019-10-14 20:06:50,347 WARN
> org.apache.hadoop.hbase.master.assignment.AssignmentManager: Failed to
> checkOnlineRegionsReport, maybe due to network lag, if this message
> continues, be careful of double assign
> org.apache.hadoop.hbase.exceptions.UnexpectedStateException: Not online:
> ,ced91598,1570663084277_0001.c727b4ed71a3a29a44cce3a917f753f4.
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.checkOnlineRegionsReport(AssignmentManager.java:1014)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportOnlineRegions(AssignmentManager.java:973)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:491)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:13139)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-23171) Ensure region state error messages include server making report (branch-2.1)
[
https://issues.apache.org/jira/browse/HBASE-23171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey resolved HBASE-23171.
-
Resolution: Fixed
> Ensure region state error messages include server making report (branch-2.1)
>
>
> Key: HBASE-23171
> URL: https://issues.apache.org/jira/browse/HBASE-23171
> Project: HBase
> Issue Type: Bug
> Components: amv2, Operability
>Affects Versions: 2.1.3
>Reporter: Michael Stack
>Assignee: Sean Busbey
>Priority: Major
> Fix For: 2.1.8
>
>
> This message is near useless. A RS is carrying a region that is not known to
> master but we don't know which server has it because message doesn't say who
> is complaining (this is fixed in 2.2.x).
> {code}
> 2019-10-14 20:06:50,347 WARN
> org.apache.hadoop.hbase.master.assignment.AssignmentManager: Failed to
> checkOnlineRegionsReport, maybe due to network lag, if this message
> continues, be careful of double assign
> org.apache.hadoop.hbase.exceptions.UnexpectedStateException: Not online:
> ,ced91598,1570663084277_0001.c727b4ed71a3a29a44cce3a917f753f4.
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.checkOnlineRegionsReport(AssignmentManager.java:1014)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportOnlineRegions(AssignmentManager.java:973)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:491)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:13139)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HBASE-23171) Ensure region state error messages include server making report (branch-2.1)
[
https://issues.apache.org/jira/browse/HBASE-23171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-23171 started by Sean Busbey.
---
> Ensure region state error messages include server making report (branch-2.1)
>
>
> Key: HBASE-23171
> URL: https://issues.apache.org/jira/browse/HBASE-23171
> Project: HBase
> Issue Type: Bug
> Components: amv2, Operability
>Affects Versions: 2.1.3
>Reporter: Michael Stack
>Assignee: Sean Busbey
>Priority: Major
> Fix For: 2.1.8
>
>
> This message is near useless. A RS is carrying a region that is not known to
> master but we don't know which server has it because message doesn't say who
> is complaining (this is fixed in 2.2.x).
> {code}
> 2019-10-14 20:06:50,347 WARN
> org.apache.hadoop.hbase.master.assignment.AssignmentManager: Failed to
> checkOnlineRegionsReport, maybe due to network lag, if this message
> continues, be careful of double assign
> org.apache.hadoop.hbase.exceptions.UnexpectedStateException: Not online:
> ,ced91598,1570663084277_0001.c727b4ed71a3a29a44cce3a917f753f4.
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.checkOnlineRegionsReport(AssignmentManager.java:1014)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportOnlineRegions(AssignmentManager.java:973)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:491)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:13139)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-23171) Ensure region state error messages include server making report (branch-2.1)
[
https://issues.apache.org/jira/browse/HBASE-23171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey reassigned HBASE-23171:
---
Assignee: Sean Busbey
> Ensure region state error messages include server making report (branch-2.1)
>
>
> Key: HBASE-23171
> URL: https://issues.apache.org/jira/browse/HBASE-23171
> Project: HBase
> Issue Type: Bug
> Components: amv2, Operability
>Affects Versions: 2.1.3
>Reporter: Michael Stack
>Assignee: Sean Busbey
>Priority: Major
> Fix For: 2.1.8
>
>
> This message is near useless. A RS is carrying a region that is not known to
> master but we don't know which server has it because message doesn't say who
> is complaining (this is fixed in 2.2.x).
> {code}
> 2019-10-14 20:06:50,347 WARN
> org.apache.hadoop.hbase.master.assignment.AssignmentManager: Failed to
> checkOnlineRegionsReport, maybe due to network lag, if this message
> continues, be careful of double assign
> org.apache.hadoop.hbase.exceptions.UnexpectedStateException: Not online:
> ,ced91598,1570663084277_0001.c727b4ed71a3a29a44cce3a917f753f4.
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.checkOnlineRegionsReport(AssignmentManager.java:1014)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportOnlineRegions(AssignmentManager.java:973)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:491)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:13139)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23171) Ensure region state error messages include server making report (branch-2.1)
[
https://issues.apache.org/jira/browse/HBASE-23171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-23171:
Component/s: Operability
amv2
> Ensure region state error messages include server making report (branch-2.1)
>
>
> Key: HBASE-23171
> URL: https://issues.apache.org/jira/browse/HBASE-23171
> Project: HBase
> Issue Type: Bug
> Components: amv2, Operability
>Affects Versions: 2.1.3
>Reporter: Michael Stack
>Priority: Major
> Fix For: 2.1.8
>
>
> This message is near useless. A RS is carrying a region that is not known to
> master but we don't know which server has it because message doesn't say who
> is complaining (this is fixed in 2.2.x).
> {code}
> 2019-10-14 20:06:50,347 WARN
> org.apache.hadoop.hbase.master.assignment.AssignmentManager: Failed to
> checkOnlineRegionsReport, maybe due to network lag, if this message
> continues, be careful of double assign
> org.apache.hadoop.hbase.exceptions.UnexpectedStateException: Not online:
> ,ced91598,1570663084277_0001.c727b4ed71a3a29a44cce3a917f753f4.
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.checkOnlineRegionsReport(AssignmentManager.java:1014)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportOnlineRegions(AssignmentManager.java:973)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:491)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:13139)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23171) Ensure region state error messages include server making report (branch-2.1)
[
https://issues.apache.org/jira/browse/HBASE-23171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-23171:
Summary: Ensure region state error messages include server making report
(branch-2.1) (was: Not online message does not state server making report in
hbase-2.1.x)
> Ensure region state error messages include server making report (branch-2.1)
>
>
> Key: HBASE-23171
> URL: https://issues.apache.org/jira/browse/HBASE-23171
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.1.3
>Reporter: Michael Stack
>Priority: Major
> Fix For: 2.1.8
>
>
> This message is near useless. A RS is carrying a region that is not known to
> master but we don't know which server has it because message doesn't say who
> is complaining (this is fixed in 2.2.x).
> {code}
> 2019-10-14 20:06:50,347 WARN
> org.apache.hadoop.hbase.master.assignment.AssignmentManager: Failed to
> checkOnlineRegionsReport, maybe due to network lag, if this message
> continues, be careful of double assign
> org.apache.hadoop.hbase.exceptions.UnexpectedStateException: Not online:
> ,ced91598,1570663084277_0001.c727b4ed71a3a29a44cce3a917f753f4.
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.checkOnlineRegionsReport(AssignmentManager.java:1014)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportOnlineRegions(AssignmentManager.java:973)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:491)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:13139)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23171) Not online message does not state server making report in hbase-2.1.x
[
https://issues.apache.org/jira/browse/HBASE-23171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962702#comment-16962702
]
Sean Busbey commented on HBASE-23171:
-
as a heads up for folks who might hit this message. it's only in branch-2.1
because the section got overhauled on later branches in HBASE-21463. it looks
like the branch-2.1 backport is still open in HBASE-21474
> Not online message does not state server making report in hbase-2.1.x
> -
>
> Key: HBASE-23171
> URL: https://issues.apache.org/jira/browse/HBASE-23171
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.1.3
>Reporter: Michael Stack
>Priority: Major
> Fix For: 2.1.8
>
>
> This message is near useless. A RS is carrying a region that is not known to
> master but we don't know which server has it because message doesn't say who
> is complaining (this is fixed in 2.2.x).
> {code}
> 2019-10-14 20:06:50,347 WARN
> org.apache.hadoop.hbase.master.assignment.AssignmentManager: Failed to
> checkOnlineRegionsReport, maybe due to network lag, if this message
> continues, be careful of double assign
> org.apache.hadoop.hbase.exceptions.UnexpectedStateException: Not online:
> ,ced91598,1570663084277_0001.c727b4ed71a3a29a44cce3a917f753f4.
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.checkOnlineRegionsReport(AssignmentManager.java:1014)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportOnlineRegions(AssignmentManager.java:973)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:491)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:13139)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-21474) Backport "HBASE-21463 The checkOnlineRegionsReport can accidentally complete a TRSP" to branch-2.1 and branch-2.0
[ https://issues.apache.org/jira/browse/HBASE-21474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-21474: Affects Version/s: 2.1.0 > Backport "HBASE-21463 The checkOnlineRegionsReport can accidentally complete > a TRSP" to branch-2.1 and branch-2.0 > - > > Key: HBASE-21474 > URL: https://issues.apache.org/jira/browse/HBASE-21474 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Affects Versions: 2.1.0 >Reporter: Duo Zhang >Priority: Major > > As discussed in HBASE-21463, the problem may also effect branch-2.1 and > branch-2.0 but the code is a bit different, at least the UT in HBASE-21463 > can not be applied to branch-2.1 and branch-2.0. Need to dig more. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23227) Upgrade jackson-databind to 2.9.10.1
[ https://issues.apache.org/jira/browse/HBASE-23227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-23227. - Fix Version/s: 2.2.3 2.1.8 2.3.0 3.0.0 Resolution: Fixed okay, now on all branches. > Upgrade jackson-databind to 2.9.10.1 > > > Key: HBASE-23227 > URL: https://issues.apache.org/jira/browse/HBASE-23227 > Project: HBase > Issue Type: Task > Components: dependencies, REST, security >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Blocker > Fix For: 3.0.0, 2.3.0, 1.4.12, 2.1.8, 1.5.1, 2.2.3 > > > Several net new CVEs were raised against jackson-databind 2.9.10. > CVE-2019-16942 > CVE-2019-16943 > 2.9.10.1 is released, which I believe addresses these two CVEs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23227) Upgrade jackson-databind to 2.9.10.1
[ https://issues.apache.org/jira/browse/HBASE-23227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-23227: Summary: Upgrade jackson-databind to 2.9.10.1 (was: Upgrade jackson-databind to 2.9.10.1 (branch-1)) > Upgrade jackson-databind to 2.9.10.1 > > > Key: HBASE-23227 > URL: https://issues.apache.org/jira/browse/HBASE-23227 > Project: HBase > Issue Type: Task > Components: dependencies, REST, security >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Blocker > Fix For: 1.4.12, 1.5.1 > > > Several net new CVEs were raised against jackson-databind 2.9.10. > CVE-2019-16942 > CVE-2019-16943 > 2.9.10.1 is released, which I believe addresses these two CVEs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-23227) Upgrade jackson-databind to 2.9.10.1 (branch-1)
[ https://issues.apache.org/jira/browse/HBASE-23227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey reopened HBASE-23227: - reopening to forward port to master and branches-2 > Upgrade jackson-databind to 2.9.10.1 (branch-1) > --- > > Key: HBASE-23227 > URL: https://issues.apache.org/jira/browse/HBASE-23227 > Project: HBase > Issue Type: Task > Components: dependencies, REST, security >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Blocker > Fix For: 1.4.12, 1.5.1 > > > Several net new CVEs were raised against jackson-databind 2.9.10. > CVE-2019-16942 > CVE-2019-16943 > 2.9.10.1 is released, which I believe addresses these two CVEs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22969) A new binary component comparator(BinaryComponentComparator) to perform comparison of arbitrary length and position
[
https://issues.apache.org/jira/browse/HBASE-22969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962286#comment-16962286
]
Sean Busbey commented on HBASE-22969:
-
please include a ref guide update on expected use of the filter.
> A new binary component comparator(BinaryComponentComparator) to perform
> comparison of arbitrary length and position
> ---
>
> Key: HBASE-22969
> URL: https://issues.apache.org/jira/browse/HBASE-22969
> Project: HBase
> Issue Type: New Feature
> Components: Filters
>Reporter: Udai Bhan Kashyap
>Assignee: Udai Bhan Kashyap
>Priority: Minor
> Attachments: HBASE-22969.0003.patch, HBASE-22969.0004.patch,
> HBASE-22969.0005.patch, HBASE-22969.0006.patch, HBASE-22969.0007.patch,
> HBASE-22969.0008.patch, HBASE-22969.0009.patch, HBASE-22969.0010.patch,
> HBASE-22969.0011.patch, HBASE-22969.HBASE-22969.0001.patch,
> HBASE-22969.master.0001.patch
>
>
> Lets say you have composite key: a+b+c+d. And for simplicity assume that
> a,b,c, and d all are 4 byte integers.
> Now, if you want to execute a query which is semantically same to following
> sql:
> {{"SELECT * from table where a=1 and b > 10 and b < 20 and c > 90 and c < 100
> and d=1"}}
> The only choice you have is to do client side filtering. That could be lots
> of unwanted data going through various software components and network.
> Solution:
> We can create a "component" comparator which takes the value of the
> "component" and its relative position in the key to pass the 'Filter'
> subsystem of the server:
> {code}
> FilterList filterList = new FilterList(FilterList.Operator.MUST_PASS_ALL);
> int bOffset = 4;
> byte[] b10 = Bytes.toBytes(10);
> Filter b10Filter = new RowFilter(CompareFilter.CompareOp.GREATER,
> new BinaryComponentComparator(b10,bOffset));
> filterList.addFilter(b10Filter);
> byte[] b20 = Bytes.toBytes(20);
> Filter b20Filter = new RowFilter(CompareFilter.CompareOp.LESS,
> new BinaryComponentComparator(b20,bOffset));
> filterList.addFilter(b20Filter);
> int cOffset = 8;
> byte[] c90 = Bytes.toBytes(90);
> Filter c90Filter = new RowFilter(CompareFilter.CompareOp.GREATER,
> new BinaryComponentComparator(c90,cOffset));
> filterList.addFilter(c90Filter);
> byte[] c100 = Bytes.toBytes(100);
> Filter c100Filter = new RowFilter(CompareFilter.CompareOp.LESS,
> new BinaryComponentComparator(c100,cOffset));
> filterList.addFilter(c100Filter);
> in dOffset = 12;
> byte[] d1 = Bytes.toBytes(1);
> Filter dFilter = new RowFilter(CompareFilter.CompareOp.EQUAL,
> new BinaryComponentComparator(d1,dOffset));
> filterList.addFilter(dFilter);
> //build start and end key for scan
> int aOffset = 0;
> byte[] startKey = new byte[16]; //key size with four ints
> Bytes.putInt(startKey,aOffset,1); //a=1
> Bytes.putInt(startKey,bOffset,11); //b=11, takes care of b > 10
> Bytes.putInt(startKey,cOffset,91); //c=91,
> Bytes.putInt(startKey,dOffset,1); //d=1,
> byte[] endKey = new byte[16];
> Bytes.putInt(endKey,aOffset,1); //a=1
> Bytes.putInt(endKey,bOffset,20); //b=20, takes care of b < 20
> Bytes.putInt(endKey,cOffset,100); //c=100,
> Bytes.putInt(endKey,dOffset,1); //d=1,
> //setup scan
> Scan scan = new Scan(startKey,endKey);
> scan.setFilter(filterList);
> //The scanner below now should give only desired rows.
> //No client side filtering is required.
> ResultScanner scanner = table.getScanner(scan);
> {code}
> The comparator can be used with any filter which makes use of
> ByteArrayComparable. Most notably it can be used with ValueFilter to filter
> out KV based on partial comparison of 'values' :
> {code}
> byte[] partialValue = Bytes.toBytes("partial_value");
> int partialValueOffset =
> Filter partialValueFilter = new
> ValueFilter(CompareFilter.CompareOp.GREATER,
> new BinaryComponentComparator(partialValue,partialValueOffset));
> {code}
> Which in turn can be combined with RowFilter to create a poweful predicate:
> {code}
> RowFilter rowFilter = new RowFilter(GREATER, new
> BinaryComponentComparator(Bytes.toBytes("a"),1);
> FilterLiost fl = new FilterList
> (MUST_PASS_ALL,rowFilter,partialValueFilter);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-22969) A new binary component comparator(BinaryComponentComparator) to perform comparison of arbitrary length and position
[
https://issues.apache.org/jira/browse/HBASE-22969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-22969:
Issue Type: New Feature (was: Improvement)
> A new binary component comparator(BinaryComponentComparator) to perform
> comparison of arbitrary length and position
> ---
>
> Key: HBASE-22969
> URL: https://issues.apache.org/jira/browse/HBASE-22969
> Project: HBase
> Issue Type: New Feature
> Components: Filters
>Reporter: Udai Bhan Kashyap
>Assignee: Udai Bhan Kashyap
>Priority: Minor
> Attachments: HBASE-22969.0003.patch, HBASE-22969.0004.patch,
> HBASE-22969.0005.patch, HBASE-22969.0006.patch, HBASE-22969.0007.patch,
> HBASE-22969.0008.patch, HBASE-22969.0009.patch, HBASE-22969.0010.patch,
> HBASE-22969.0011.patch, HBASE-22969.HBASE-22969.0001.patch,
> HBASE-22969.master.0001.patch
>
>
> Lets say you have composite key: a+b+c+d. And for simplicity assume that
> a,b,c, and d all are 4 byte integers.
> Now, if you want to execute a query which is semantically same to following
> sql:
> {{"SELECT * from table where a=1 and b > 10 and b < 20 and c > 90 and c < 100
> and d=1"}}
> The only choice you have is to do client side filtering. That could be lots
> of unwanted data going through various software components and network.
> Solution:
> We can create a "component" comparator which takes the value of the
> "component" and its relative position in the key to pass the 'Filter'
> subsystem of the server:
> {code}
> FilterList filterList = new FilterList(FilterList.Operator.MUST_PASS_ALL);
> int bOffset = 4;
> byte[] b10 = Bytes.toBytes(10);
> Filter b10Filter = new RowFilter(CompareFilter.CompareOp.GREATER,
> new BinaryComponentComparator(b10,bOffset));
> filterList.addFilter(b10Filter);
> byte[] b20 = Bytes.toBytes(20);
> Filter b20Filter = new RowFilter(CompareFilter.CompareOp.LESS,
> new BinaryComponentComparator(b20,bOffset));
> filterList.addFilter(b20Filter);
> int cOffset = 8;
> byte[] c90 = Bytes.toBytes(90);
> Filter c90Filter = new RowFilter(CompareFilter.CompareOp.GREATER,
> new BinaryComponentComparator(c90,cOffset));
> filterList.addFilter(c90Filter);
> byte[] c100 = Bytes.toBytes(100);
> Filter c100Filter = new RowFilter(CompareFilter.CompareOp.LESS,
> new BinaryComponentComparator(c100,cOffset));
> filterList.addFilter(c100Filter);
> in dOffset = 12;
> byte[] d1 = Bytes.toBytes(1);
> Filter dFilter = new RowFilter(CompareFilter.CompareOp.EQUAL,
> new BinaryComponentComparator(d1,dOffset));
> filterList.addFilter(dFilter);
> //build start and end key for scan
> int aOffset = 0;
> byte[] startKey = new byte[16]; //key size with four ints
> Bytes.putInt(startKey,aOffset,1); //a=1
> Bytes.putInt(startKey,bOffset,11); //b=11, takes care of b > 10
> Bytes.putInt(startKey,cOffset,91); //c=91,
> Bytes.putInt(startKey,dOffset,1); //d=1,
> byte[] endKey = new byte[16];
> Bytes.putInt(endKey,aOffset,1); //a=1
> Bytes.putInt(endKey,bOffset,20); //b=20, takes care of b < 20
> Bytes.putInt(endKey,cOffset,100); //c=100,
> Bytes.putInt(endKey,dOffset,1); //d=1,
> //setup scan
> Scan scan = new Scan(startKey,endKey);
> scan.setFilter(filterList);
> //The scanner below now should give only desired rows.
> //No client side filtering is required.
> ResultScanner scanner = table.getScanner(scan);
> {code}
> The comparator can be used with any filter which makes use of
> ByteArrayComparable. Most notably it can be used with ValueFilter to filter
> out KV based on partial comparison of 'values' :
> {code}
> byte[] partialValue = Bytes.toBytes("partial_value");
> int partialValueOffset =
> Filter partialValueFilter = new
> ValueFilter(CompareFilter.CompareOp.GREATER,
> new BinaryComponentComparator(partialValue,partialValueOffset));
> {code}
> Which in turn can be combined with RowFilter to create a poweful predicate:
> {code}
> RowFilter rowFilter = new RowFilter(GREATER, new
> BinaryComponentComparator(Bytes.toBytes("a"),1);
> FilterLiost fl = new FilterList
> (MUST_PASS_ALL,rowFilter,partialValueFilter);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HBASE-23229) Update branch-1 to 1.6.0-SNAPSHOT
[ https://issues.apache.org/jira/browse/HBASE-23229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-23229 started by Sean Busbey. --- > Update branch-1 to 1.6.0-SNAPSHOT > - > > Key: HBASE-23229 > URL: https://issues.apache.org/jira/browse/HBASE-23229 > Project: HBase > Issue Type: Task > Components: build, community >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > > since 1.5.0 was released from branch-1 we've had new feature changes land. > update the branch so the next release from it will be 1.6.0. > subsequent 1.5.z releases can be made by creating a new branch from rel/1.5.0 > and cherry-picking desired changes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23229) Update branch-1 to 1.6.0-SNAPSHOT
Sean Busbey created HBASE-23229: --- Summary: Update branch-1 to 1.6.0-SNAPSHOT Key: HBASE-23229 URL: https://issues.apache.org/jira/browse/HBASE-23229 Project: HBase Issue Type: Task Components: build, community Reporter: Sean Busbey since 1.5.0 was released from branch-1 we've had new feature changes land. update the branch so the next release from it will be 1.6.0. subsequent 1.5.z releases can be made by creating a new branch from rel/1.5.0 and cherry-picking desired changes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-23229) Update branch-1 to 1.6.0-SNAPSHOT
[ https://issues.apache.org/jira/browse/HBASE-23229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey reassigned HBASE-23229: --- Assignee: Sean Busbey > Update branch-1 to 1.6.0-SNAPSHOT > - > > Key: HBASE-23229 > URL: https://issues.apache.org/jira/browse/HBASE-23229 > Project: HBase > Issue Type: Task > Components: build, community >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > > since 1.5.0 was released from branch-1 we've had new feature changes land. > update the branch so the next release from it will be 1.6.0. > subsequent 1.5.z releases can be made by creating a new branch from rel/1.5.0 > and cherry-picking desired changes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23208) Unit formatting in Master & RS UI
[ https://issues.apache.org/jira/browse/HBASE-23208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-23208: Fix Version/s: 1.5.1 > Unit formatting in Master & RS UI > - > > Key: HBASE-23208 > URL: https://issues.apache.org/jira/browse/HBASE-23208 > Project: HBase > Issue Type: Improvement > Components: UI >Affects Versions: 3.0.0 >Reporter: Karthik Palanisamy >Assignee: Karthik Palanisamy >Priority: Trivial > Fix For: 3.0.0, 2.3.0, 1.5.1 > > Attachments: Screen Shot 2019-10-23 at 2.01.44 PM.png, Screen Shot > 2019-10-23 at 2.35.04 PM.png > > > ProcV2 use > org.apache.hadoop.hbase.procedure2.util.StringUtils#humanTimeDiff(long) and > humanSize(double) where it returns human readable string. I think it would > good if we format the string before we return it. > > !Screen Shot 2019-10-23 at 2.01.44 PM.png! > !Screen Shot 2019-10-23 at 2.35.04 PM.png! > > the same format will apply to master and regionserver logs. I hope no one > will concern about this format change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23208) Unit formatting in Master & RS UI
[ https://issues.apache.org/jira/browse/HBASE-23208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-23208: Fix Version/s: (was: 1.5.1) 1.6.0 > Unit formatting in Master & RS UI > - > > Key: HBASE-23208 > URL: https://issues.apache.org/jira/browse/HBASE-23208 > Project: HBase > Issue Type: Improvement > Components: UI >Affects Versions: 3.0.0 >Reporter: Karthik Palanisamy >Assignee: Karthik Palanisamy >Priority: Trivial > Fix For: 3.0.0, 2.3.0, 1.6.0 > > Attachments: Screen Shot 2019-10-23 at 2.01.44 PM.png, Screen Shot > 2019-10-23 at 2.35.04 PM.png > > > ProcV2 use > org.apache.hadoop.hbase.procedure2.util.StringUtils#humanTimeDiff(long) and > humanSize(double) where it returns human readable string. I think it would > good if we format the string before we return it. > > !Screen Shot 2019-10-23 at 2.01.44 PM.png! > !Screen Shot 2019-10-23 at 2.35.04 PM.png! > > the same format will apply to master and regionserver logs. I hope no one > will concern about this format change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23208) Unit formatting in Master & RS UI
[ https://issues.apache.org/jira/browse/HBASE-23208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-23208. - Fix Version/s: 2.3.0 3.0.0 Resolution: Fixed thanks for the fixup [~kpalanisamy]! > Unit formatting in Master & RS UI > - > > Key: HBASE-23208 > URL: https://issues.apache.org/jira/browse/HBASE-23208 > Project: HBase > Issue Type: Improvement > Components: UI >Affects Versions: 3.0.0 >Reporter: Karthik Palanisamy >Assignee: Karthik Palanisamy >Priority: Trivial > Fix For: 3.0.0, 2.3.0 > > Attachments: Screen Shot 2019-10-23 at 2.01.44 PM.png, Screen Shot > 2019-10-23 at 2.35.04 PM.png > > > ProcV2 use > org.apache.hadoop.hbase.procedure2.util.StringUtils#humanTimeDiff(long) and > humanSize(double) where it returns human readable string. I think it would > good if we format the string before we return it. > > !Screen Shot 2019-10-23 at 2.01.44 PM.png! > !Screen Shot 2019-10-23 at 2.35.04 PM.png! > > the same format will apply to master and regionserver logs. I hope no one > will concern about this format change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23227) Upgrade jackson-databind to 2.9.10.1 (branch-1)
[ https://issues.apache.org/jira/browse/HBASE-23227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-23227. - Fix Version/s: 1.5.1 1.4.12 Release Note: the Apache HBase REST Proxy now uses Jackson Databind version 2.9.10.1 to address the following CVEs - CVE-2019-16942 - CVE-2019-16943 Users of prior releases with Jackson Databind 2.9.10 are advised to either upgrade to this release or to upgrade their local Jackson Databind jar directly. Resolution: Fixed > Upgrade jackson-databind to 2.9.10.1 (branch-1) > --- > > Key: HBASE-23227 > URL: https://issues.apache.org/jira/browse/HBASE-23227 > Project: HBase > Issue Type: Task > Components: dependencies, REST, security >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Blocker > Fix For: 1.4.12, 1.5.1 > > > Several net new CVEs were raised against jackson-databind 2.9.10. > CVE-2019-16942 > CVE-2019-16943 > 2.9.10.1 is released, which I believe addresses these two CVEs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23228) Allow for jdk8 specific modules on branch-1 in precommit/nightly testing
Sean Busbey created HBASE-23228: --- Summary: Allow for jdk8 specific modules on branch-1 in precommit/nightly testing Key: HBASE-23228 URL: https://issues.apache.org/jira/browse/HBASE-23228 Project: HBase Issue Type: Improvement Components: build, test Reporter: Sean Busbey Assignee: Sean Busbey At least 1 feature backport is waiting on proper handling of jdk8 activated modules for our yetus personality (HBASE-22114 tinylfu). Implement the general handling here so that we don't have to worry about pushes to the PR branch overwriting it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23227) Upgrade jackson-databind to 2.9.10.1 (branch-1)
[ https://issues.apache.org/jira/browse/HBASE-23227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961997#comment-16961997 ] Sean Busbey commented on HBASE-23227: - I'm going to go ahead and merge this and then look for impact in nightly. [~weichiu] could you like the master/branch-2 version as related or just note here that those versions are not impacted for some reason (I don't remember off hand if we're still using jackson for the rest server). > Upgrade jackson-databind to 2.9.10.1 (branch-1) > --- > > Key: HBASE-23227 > URL: https://issues.apache.org/jira/browse/HBASE-23227 > Project: HBase > Issue Type: Task > Components: dependencies, REST, security >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Blocker > > Several net new CVEs were raised against jackson-databind 2.9.10. > CVE-2019-16942 > CVE-2019-16943 > 2.9.10.1 is released, which I believe addresses these two CVEs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23225) Error building shaded-client: duplicate entry: META-INF/.../ObjectCodec
[
https://issues.apache.org/jira/browse/HBASE-23225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961382#comment-16961382
]
Sean Busbey commented on HBASE-23225:
-
you have my sympathies. glad it worked out.
> Error building shaded-client: duplicate entry: META-INF/.../ObjectCodec
>
>
> Key: HBASE-23225
> URL: https://issues.apache.org/jira/browse/HBASE-23225
> Project: HBase
> Issue Type: Bug
> Environment: $ mvn --version
> Apache Maven 3.6.2 (40f52333136460af0dc0d7232c0dc0bcf0d9e117;
> 2019-08-27T08:06:16-07:00)
> Maven home: /usr/local/Cellar/maven/3.6.2/libexec
> Java version: 1.8.0_232, vendor: Azul Systems, Inc., runtime:
> /Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "mac os x", version: "10.14.6", arch: "x86_64", family: "mac"
>Reporter: Andrew Kyle Purtell
>Priority: Major
>
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.1.1:shade
> (aggregate-into-a-jar-with-relocated-third-parties) on project
> hbase-shaded-client:
> Error creating shaded jar:
> duplicate entry:
> META-INF/services/org.apache.hadoop.hbase.shaded.com.fasterxml.jackson.core.ObjectCodec
>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23225) Error building shaded-client: duplicate entry: META-INF/.../ObjectCodec
[
https://issues.apache.org/jira/browse/HBASE-23225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961273#comment-16961273
]
Sean Busbey commented on HBASE-23225:
-
wth. we should already be doing this via
[ServicesResourceTransformer|https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer].
Can you attach a maven log?
> Error building shaded-client: duplicate entry: META-INF/.../ObjectCodec
>
>
> Key: HBASE-23225
> URL: https://issues.apache.org/jira/browse/HBASE-23225
> Project: HBase
> Issue Type: Bug
> Environment: $ mvn --version
> Apache Maven 3.6.2 (40f52333136460af0dc0d7232c0dc0bcf0d9e117;
> 2019-08-27T08:06:16-07:00)
> Maven home: /usr/local/Cellar/maven/3.6.2/libexec
> Java version: 1.8.0_232, vendor: Azul Systems, Inc., runtime:
> /Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "mac os x", version: "10.14.6", arch: "x86_64", family: "mac"
>Reporter: Andrew Kyle Purtell
>Priority: Major
>
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.1.1:shade
> (aggregate-into-a-jar-with-relocated-third-parties) on project
> hbase-shaded-client:
> Error creating shaded jar:
> duplicate entry:
> META-INF/services/org.apache.hadoop.hbase.shaded.com.fasterxml.jackson.core.ObjectCodec
>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23225) Error building shaded-client: duplicate entry: META-INF/.../ObjectCodec
[
https://issues.apache.org/jira/browse/HBASE-23225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961258#comment-16961258
]
Sean Busbey commented on HBASE-23225:
-
is there a build warning about us not setting a shade plugin or anything? I'm
wondering why this behavior would have changed recently.
> Error building shaded-client: duplicate entry: META-INF/.../ObjectCodec
>
>
> Key: HBASE-23225
> URL: https://issues.apache.org/jira/browse/HBASE-23225
> Project: HBase
> Issue Type: Bug
> Environment: $ mvn --version
> Apache Maven 3.6.2 (40f52333136460af0dc0d7232c0dc0bcf0d9e117;
> 2019-08-27T08:06:16-07:00)
> Maven home: /usr/local/Cellar/maven/3.6.2/libexec
> Java version: 1.8.0_232, vendor: Azul Systems, Inc., runtime:
> /Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "mac os x", version: "10.14.6", arch: "x86_64", family: "mac"
>Reporter: Andrew Kyle Purtell
>Priority: Major
>
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.1.1:shade
> (aggregate-into-a-jar-with-relocated-third-parties) on project
> hbase-shaded-client:
> Error creating shaded jar:
> duplicate entry:
> META-INF/services/org.apache.hadoop.hbase.shaded.com.fasterxml.jackson.core.ObjectCodec
>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23225) Error building shaded-client: duplicate entry: META-INF/.../ObjectCodec
[
https://issues.apache.org/jira/browse/HBASE-23225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961254#comment-16961254
]
Sean Busbey commented on HBASE-23225:
-
ugh. maintenance tail coming to bite us again. AFAIK we never did the work to
make sure we only stuff from one place.
but! some luck that this is specifically the services definition. there is
already a plugin that will aggregate this for us. lemme find it. (I'm not sure
if this is the "correct" correct solution, since our internal use should be
under a different package. I think this means we're getting two from different
hadoop dependencies?)
lemme try to find it quickly
> Error building shaded-client: duplicate entry: META-INF/.../ObjectCodec
>
>
> Key: HBASE-23225
> URL: https://issues.apache.org/jira/browse/HBASE-23225
> Project: HBase
> Issue Type: Bug
> Environment: $ mvn --version
> Apache Maven 3.6.2 (40f52333136460af0dc0d7232c0dc0bcf0d9e117;
> 2019-08-27T08:06:16-07:00)
> Maven home: /usr/local/Cellar/maven/3.6.2/libexec
> Java version: 1.8.0_232, vendor: Azul Systems, Inc., runtime:
> /Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "mac os x", version: "10.14.6", arch: "x86_64", family: "mac"
>Reporter: Andrew Kyle Purtell
>Priority: Major
>
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.1.1:shade
> (aggregate-into-a-jar-with-relocated-third-parties) on project
> hbase-shaded-client:
> Error creating shaded jar:
> duplicate entry:
> META-INF/services/org.apache.hadoop.hbase.shaded.com.fasterxml.jackson.core.ObjectCodec
>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-23222) Better logging and mitigation for MOB compaction failures
[ https://issues.apache.org/jira/browse/HBASE-23222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-23222. - Fix Version/s: 2.2.3 2.1.8 2.3.0 3.0.0 Release Note: The MOB compaction process in the HBase Master now logs more about its activity. In the event that you run into the problems described in HBASE-22075, there is a new HFileCleanerDelegate that will stop all removal of MOB hfiles from the archive area. It can be configured by adding `org.apache.hadoop.hbase.mob.ManualMobMaintHFileCleaner` to the list configured for `hbase.master.hfilecleaner.plugins`. This new cleaner delegate will cause your archive area to grow unbounded; you will have to manually prune files which may be prohibitively complex. Consider if your use case will allow you to mitigate by disabling mob compactions instead. Caveats: * Be sure the list of cleaner delegates still includes the default cleaners you will likely need: ttl, snapshot, and hlink. * Be mindful that if you enable this cleaner delegate then there will be *no* automated process for removing these mob hfiles. You should see a single region per table in `%hbase_root%/archive` that accumulates files over time. You will have to determine which of these files are safe or not to remove. * You should list this cleaner delegate after the snapshot and hlink delegates so that you can enable sufficient logging to determine when an archived mob hfile is needed by those subsystems. When set to `TRACE` logging, the CleanerChore logger will include archive retention decision justifications. * If your use case creates a large number of uniquely named tables, this new delegate will cause memory pressure on the master. Resolution: Fixed > Better logging and mitigation for MOB compaction failures > - > > Key: HBASE-23222 > URL: https://issues.apache.org/jira/browse/HBASE-23222 > Project: HBase > Issue Type: Bug > Components: mob >Affects Versions: 2.1.0, 2.0.0, 2.2.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3 > > > Some logging and mitigation options for MOB dataloss issues described in > HBASE-22075. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-23194) Remove unused methods from TokenUtil
[ https://issues.apache.org/jira/browse/HBASE-23194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey reopened HBASE-23194: - Please revert the removal of methods in TokenUtil that were deprecated in HBASE-22027. TokenUtil is still IA.Public and those deprecation warnings haven't even made it into a HBase 2.y release yet. They shouldn't be removed until HBase 4.0. > Remove unused methods from TokenUtil > > > Key: HBASE-23194 > URL: https://issues.apache.org/jira/browse/HBASE-23194 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Fix For: 3.0.0 > > > Cleanup TokenUtil: remove unused methods from TokenUtil. For util methods to > obtain Authentication tokens, ClientTokenUtil should be used where possible > (in absence of hbase-server dependency) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22640) Random init hstore lastFlushTime
[ https://issues.apache.org/jira/browse/HBASE-22640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961144#comment-16961144 ] Sean Busbey commented on HBASE-22640: - any update? this ready to go in? > Random init hstore lastFlushTime > - > > Key: HBASE-22640 > URL: https://issues.apache.org/jira/browse/HBASE-22640 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Bing Xiao >Assignee: Bing Xiao >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3 > > Attachments: HBASE-22640-master-v1.patch > > > During with open region use current time as hstore last flush time, and no > mush data put cause memstore flush, after flushCheckInterval all memstore > will flush together bring concentrated IO and compaction make high request > latency；So random init lastFlushTime -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23082) Backport low-latency snapshot tracking for space quotas to 2.x
[ https://issues.apache.org/jira/browse/HBASE-23082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961143#comment-16961143 ] Sean Busbey commented on HBASE-23082: - +1. I'd rather see the checkstyle and whitespace checks pass, but the reports have aged off so I don't feel strongly. > Backport low-latency snapshot tracking for space quotas to 2.x > -- > > Key: HBASE-23082 > URL: https://issues.apache.org/jira/browse/HBASE-23082 > Project: HBase > Issue Type: Improvement > Components: Quotas >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 2.3.0, 2.1.8, 2.2.3 > > Attachments: HBASE-23082.001.patch, HBASE-23082.002.branch-2.patch, > HBASE-23082.002.patch, HBASE-23082.branch-2.1.002.patch > > > Some rebase'ing at $dayjob found that HBASE-18133 and HBASE-18135 never made > it to any branch-2's. > There's no good reason that I can think of as to why we shouldn't backport > these (will make quotas and snapshots much more accurate). As memory serves > me, they just were landing when we were branching for early 2.0.0 releases. > Let's get backports for HBASE-18133, HBASE-18135, and HBASE-20531 (and > implicitly HBASE-20439 and HBASE-20440) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23222) Better logging and mitigation for MOB compaction failures
Sean Busbey created HBASE-23222: --- Summary: Better logging and mitigation for MOB compaction failures Key: HBASE-23222 URL: https://issues.apache.org/jira/browse/HBASE-23222 Project: HBase Issue Type: Bug Components: mob Affects Versions: 2.2.0, 2.0.0, 2.1.0 Reporter: Sean Busbey Assignee: Sean Busbey Some logging and mitigation options for MOB dataloss issues described in HBASE-22075. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23220) Release 1.5.1
[ https://issues.apache.org/jira/browse/HBASE-23220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16960691#comment-16960691 ] Sean Busbey commented on HBASE-23220: - both tests look solid now since the revert. > Release 1.5.1 > - > > Key: HBASE-23220 > URL: https://issues.apache.org/jira/browse/HBASE-23220 > Project: HBase > Issue Type: Task > Components: community >Affects Versions: 1.5.1 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 1.5.1 > > > let's roll 1.5.1 to get HBASE-23174 out on 1.5.z. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23220) Release 1.5.1
[ https://issues.apache.org/jira/browse/HBASE-23220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16960265#comment-16960265 ] Sean Busbey commented on HBASE-23220: - tracked those down to HBASE-23185 and reverted it. both tests now pass locally. started some flaky test runs to confirm. > Release 1.5.1 > - > > Key: HBASE-23220 > URL: https://issues.apache.org/jira/browse/HBASE-23220 > Project: HBase > Issue Type: Task > Components: community >Affects Versions: 1.5.1 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 1.5.1 > > > let's roll 1.5.1 to get HBASE-23174 out on 1.5.z. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23185) High cpu usage because getTable()#put() gets config value every time
[ https://issues.apache.org/jira/browse/HBASE-23185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16960263#comment-16960263 ] Sean Busbey commented on HBASE-23185: - I reverted this from branch-1 because it's causing TestMasterNoCluster and TestCatalogJanitor to fail there. > High cpu usage because getTable()#put() gets config value every time > > > Key: HBASE-23185 > URL: https://issues.apache.org/jira/browse/HBASE-23185 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 1.5.0, 1.4.10, 1.2.12, 1.3.5 >Reporter: Shinya Yoshida >Assignee: Shinya Yoshida >Priority: Major > Labels: performance > Fix For: 1.6.0 > > Attachments: Screenshot from 2019-10-18 12-38-14.png, Screenshot from > 2019-10-18 13-03-24.png > > > When we analyzed the performance of our hbase application with many puts, we > found that Configuration methods use many CPU resources: > !Screenshot from 2019-10-18 12-38-14.png|width=460,height=205! > As you can see, getTable().put() is calling Configuration methods which cause > regex or synchronization by Hashtable. > This should not happen in 0.99.2 because > https://issues.apache.org/jira/browse/HBASE-12128 addressed such an issue. > However, it's reproducing nowadays by bugs or leakages after many code > evoluations between 0.9x and 1.x. > # > [https://github.com/apache/hbase/blob/dd9eadb00f9dcd071a246482a11dfc7d63845f00/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java#L369-L374] > ** finishSetup is called every new HTable() e.g. every con.getTable() > ** So getInt is called everytime and it does regex > # > [https://github.com/apache/hbase/blob/dd9eadb00f9dcd071a246482a11dfc7d63845f00/hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutatorImpl.java#L115] > ** BufferedMutatorImpl is created every first put for HTable e.g. > con.getTable().put() > ** Create ConnectionConf every time in BufferedMutatorImpl constructor > ** ConnectionConf gets config value in the constructor > # > [https://github.com/apache/hbase/blob/dd9eadb00f9dcd071a246482a11dfc7d63845f00/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java#L326] > ** AsyncProcess is created in BufferedMutatorImpl constructor, so new > AsyncProcess is created by con.getTable().put() > ** AsyncProcess parse many configurations > So, con.getTable().put() is heavy operation for CPU because of getting config > value. > > With in-house patch for this issue, we observed about 10% improvement on > max-throughput (e.g. CPU usage) at client-side: > !Screenshot from 2019-10-18 13-03-24.png|width=508,height=223! > > Seems branch-2 is not affected because client implementation has been changed > dramatically. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23220) Release 1.5.1
[
https://issues.apache.org/jira/browse/HBASE-23220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16960258#comment-16960258
]
Sean Busbey commented on HBASE-23220:
-
current flaky looks unacceptable:
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-1/564/artifact/dashboard.html
In paticular {{TestMasterNoCluster}} and {{TestCatalogJanitor}} are failing all
the time.
> Release 1.5.1
> -
>
> Key: HBASE-23220
> URL: https://issues.apache.org/jira/browse/HBASE-23220
> Project: HBase
> Issue Type: Task
> Components: community
>Affects Versions: 1.5.1
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Fix For: 1.5.1
>
>
> let's roll 1.5.1 to get HBASE-23174 out on 1.5.z.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HBASE-23220) Release 1.5.1
[ https://issues.apache.org/jira/browse/HBASE-23220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-23220 started by Sean Busbey. --- > Release 1.5.1 > - > > Key: HBASE-23220 > URL: https://issues.apache.org/jira/browse/HBASE-23220 > Project: HBase > Issue Type: Task > Components: community >Affects Versions: 1.5.1 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 1.5.1 > > > let's roll 1.5.1 to get HBASE-23174 out on 1.5.z. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23220) Release 1.5.1
[ https://issues.apache.org/jira/browse/HBASE-23220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16960257#comment-16960257 ] Sean Busbey commented on HBASE-23220: - moved things out to 1.5.2 in jira. > Release 1.5.1 > - > > Key: HBASE-23220 > URL: https://issues.apache.org/jira/browse/HBASE-23220 > Project: HBase > Issue Type: Task > Components: community >Affects Versions: 1.5.1 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 1.5.1 > > > let's roll 1.5.1 to get HBASE-23174 out on 1.5.z. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23220) Release 1.5.1
Sean Busbey created HBASE-23220: --- Summary: Release 1.5.1 Key: HBASE-23220 URL: https://issues.apache.org/jira/browse/HBASE-23220 Project: HBase Issue Type: Task Components: community Affects Versions: 1.5.1 Reporter: Sean Busbey Assignee: Sean Busbey Fix For: 1.5.1 let's roll 1.5.1 to get HBASE-23174 out on 1.5.z. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-22991) Release 1.4.11
[ https://issues.apache.org/jira/browse/HBASE-22991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-22991. - Resolution: Fixed website changes were merged, the updated website was published, and the release email is now up: https://lists.apache.org/thread.html/9e5b43ad54ceaa21ca68d3f00e1e092e083ab2105cf84876e408d7b3@%3Cuser.hbase.apache.org%3E > Release 1.4.11 > -- > > Key: HBASE-22991 > URL: https://issues.apache.org/jira/browse/HBASE-22991 > Project: HBase > Issue Type: Task > Components: community >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 1.4.11 > > Attachments: Flaky_20Test_20Report.zip > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22991) Release 1.4.11
[ https://issues.apache.org/jira/browse/HBASE-22991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959405#comment-16959405 ] Sean Busbey commented on HBASE-22991: - left to do AFAICT: * PR 756 * announce email to dev@ and user@ > Release 1.4.11 > -- > > Key: HBASE-22991 > URL: https://issues.apache.org/jira/browse/HBASE-22991 > Project: HBase > Issue Type: Task > Components: community >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 1.4.11 > > Attachments: Flaky_20Test_20Report.zip > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22991) Release 1.4.11
[
https://issues.apache.org/jira/browse/HBASE-22991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959403#comment-16959403
]
Sean Busbey commented on HBASE-22991:
-
for whatever reason we haven't previously published HBase 1.4 release docs, so
I added them.
{code}
hbase-site busbey$ mkdir 1.4
hbase-site busbey$ tar --strip-components=2 -xzf
~/projects/hbase_project/hbase-releases/hbase-1.4.11/hbase-1.4.11-bin.tar.gz -C
1.4/ hbase-1.4.11/docs/
hbase-site busbey$ git add 1.4
hbase-site busbey$ git commit -m "HBASE-22991 Add 1.4 release docs."
hbase-site busbey$ git commit --allow-empty -m "INFRA-10751 Empty commit"
hbase-site busbey$ git push origin asf-site
{code}
(which works for 1.4 because we still include all of the docs in the binary
tarball on that release line)
> Release 1.4.11
> --
>
> Key: HBASE-22991
> URL: https://issues.apache.org/jira/browse/HBASE-22991
> Project: HBase
> Issue Type: Task
> Components: community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Fix For: 1.4.11
>
> Attachments: Flaky_20Test_20Report.zip
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22991) Release 1.4.11
[ https://issues.apache.org/jira/browse/HBASE-22991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959396#comment-16959396 ] Sean Busbey commented on HBASE-22991: - put up PR for website update: https://github.com/apache/hbase/pull/756 > Release 1.4.11 > -- > > Key: HBASE-22991 > URL: https://issues.apache.org/jira/browse/HBASE-22991 > Project: HBase > Issue Type: Task > Components: community >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 1.4.11 > > Attachments: Flaky_20Test_20Report.zip > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22991) Release 1.4.11
[ https://issues.apache.org/jira/browse/HBASE-22991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959395#comment-16959395 ] Sean Busbey commented on HBASE-22991: - pushed the rel/1.4.11 tag > Release 1.4.11 > -- > > Key: HBASE-22991 > URL: https://issues.apache.org/jira/browse/HBASE-22991 > Project: HBase > Issue Type: Task > Components: community >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 1.4.11 > > Attachments: Flaky_20Test_20Report.zip > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22991) Release 1.4.11
[ https://issues.apache.org/jira/browse/HBASE-22991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959378#comment-16959378 ] Sean Busbey commented on HBASE-22991: - updated branch-1.4 for release. > Release 1.4.11 > -- > > Key: HBASE-22991 > URL: https://issues.apache.org/jira/browse/HBASE-22991 > Project: HBase > Issue Type: Task > Components: community >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 1.4.11 > > Attachments: Flaky_20Test_20Report.zip > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22991) Release 1.4.11
[ https://issues.apache.org/jira/browse/HBASE-22991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959369#comment-16959369 ] Sean Busbey commented on HBASE-22991: - marked 1.4.11 in jira as released. > Release 1.4.11 > -- > > Key: HBASE-22991 > URL: https://issues.apache.org/jira/browse/HBASE-22991 > Project: HBase > Issue Type: Task > Components: community >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 1.4.11 > > Attachments: Flaky_20Test_20Report.zip > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22991) Release 1.4.11
[ https://issues.apache.org/jira/browse/HBASE-22991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959367#comment-16959367 ] Sean Busbey commented on HBASE-22991: - asf nexus staging repository released > Release 1.4.11 > -- > > Key: HBASE-22991 > URL: https://issues.apache.org/jira/browse/HBASE-22991 > Project: HBase > Issue Type: Task > Components: community >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 1.4.11 > > Attachments: Flaky_20Test_20Report.zip > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
