[jira] [Updated] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21258: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.2.0 1.5.0 3.0.0 Status: Resolved (was: Patch Available) Thanks for the reviews. > Add resetting of flags for RS Group pre/post hooks in TestRSGroups > -- > > Key: HBASE-21258 > URL: https://issues.apache.org/jira/browse/HBASE-21258 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.2.0 > > Attachments: 21258.branch-1.04.txt, 21258.branch-1.05.txt, > 21258.branch-2.v1.patch, 21258.v1.txt > > > Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS > Group pre/post hooks in TestRSGroups was absent. > This issue is to add the resetting of these flags before each subtest starts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-9924) Upgrade zookeeper to 3.4.13
[ https://issues.apache.org/jira/browse/FLINK-9924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated FLINK-9924: -- Description: zookeeper 3.4.13 is being released. ZOOKEEPER-2959 fixes data loss when observer is used ZOOKEEPER-2184 allows ZooKeeper Java clients to work in dynamic IP (container / cloud) environment was: zookeeper 3.4.13 is being released. ZOOKEEPER-2959 fixes data loss when observer is used ZOOKEEPER-2184 allows ZooKeeper Java clients to work in dynamic IP (container / cloud) environment > Upgrade zookeeper to 3.4.13 > --- > > Key: FLINK-9924 > URL: https://issues.apache.org/jira/browse/FLINK-9924 > Project: Flink > Issue Type: Task >Reporter: Ted Yu >Assignee: vinoyang >Priority: Major > > zookeeper 3.4.13 is being released. > ZOOKEEPER-2959 fixes data loss when observer is used > ZOOKEEPER-2184 allows ZooKeeper Java clients to work in dynamic IP (container > / cloud) environment -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-10391) MillisOfDay is used in place of instant for LocalTime ctor in AvroKryoSerializerUtils
[
https://issues.apache.org/jira/browse/FLINK-10391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated FLINK-10391:
---
Description:
>From the JodaLocalTimeSerializer#write, we serialize getMillisOfDay() value
>from LocalTime.
For read method:
{code}
final int time = input.readInt(true);
return new LocalTime(time,
ISOChronology.getInstanceUTC().withZone(DateTimeZone.UTC));
{code}
It seems
http://joda-time.sourceforge.net/apidocs/org/joda/time/LocalTime.html#fromMillisOfDay(long,%20org.joda.time.Chronology)
should be used instead.
was:
>From the JodaLocalTimeSerializer#write, we serialize getMillisOfDay() value
>from LocalTime.
For read method:
{code}
final int time = input.readInt(true);
return new LocalTime(time,
ISOChronology.getInstanceUTC().withZone(DateTimeZone.UTC));
{code}
It seems
http://joda-time.sourceforge.net/apidocs/org/joda/time/LocalTime.html#fromMillisOfDay(long,%20org.joda.time.Chronology)
should be used instead.
> MillisOfDay is used in place of instant for LocalTime ctor in
> AvroKryoSerializerUtils
> -
>
> Key: FLINK-10391
> URL: https://issues.apache.org/jira/browse/FLINK-10391
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>
> From the JodaLocalTimeSerializer#write, we serialize getMillisOfDay() value
> from LocalTime.
> For read method:
> {code}
> final int time = input.readInt(true);
> return new LocalTime(time,
> ISOChronology.getInstanceUTC().withZone(DateTimeZone.UTC));
> {code}
> It seems
> http://joda-time.sourceforge.net/apidocs/org/joda/time/LocalTime.html#fromMillisOfDay(long,%20org.joda.time.Chronology)
> should be used instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3556) Interned string should not be used as lock object
[
https://issues.apache.org/jira/browse/KYLIN-3556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated KYLIN-3556:
--
Description:
In JDBCResourceDAO :
{code}
public void execute(Connection connection) throws SQLException {
synchronized (resPath.intern()) {
{code}
Locking on an interned string can cause unexpected locking collisions with
other part of code.
was:
In JDBCResourceDAO :
{code}
public void execute(Connection connection) throws SQLException {
synchronized (resPath.intern()) {
{code}
Locking on an interned string can cause unexpected locking collisions with
other part of code.
> Interned string should not be used as lock object
> -
>
> Key: KYLIN-3556
> URL: https://issues.apache.org/jira/browse/KYLIN-3556
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.5.0
>Reporter: Ted Yu
>Assignee: Kaige Liu
>Priority: Minor
> Fix For: v2.5.1
>
>
> In JDBCResourceDAO :
> {code}
> public void execute(Connection connection) throws SQLException {
> synchronized (resPath.intern()) {
> {code}
> Locking on an interned string can cause unexpected locking collisions with
> other part of code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3290) Avoid calling Class#newInstance
[ https://issues.apache.org/jira/browse/KYLIN-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3290: -- Description: Class#newInstance is deprecated starting in Java 9 - https://bugs.openjdk.java.net/browse/JDK-6850612 - because it may throw undeclared checked exceptions. The suggested replacement is getDeclaredConstructor().newInstance(), which wraps the checked exceptions in InvocationException. was: Class#newInstance is deprecated starting in Java 9 - https://bugs.openjdk.java.net/browse/JDK-6850612 - because it may throw undeclared checked exceptions. The suggested replacement is getDeclaredConstructor().newInstance(), which wraps the checked exceptions in InvocationException. > Avoid calling Class#newInstance > --- > > Key: KYLIN-3290 > URL: https://issues.apache.org/jira/browse/KYLIN-3290 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Assignee: jiatao.tao >Priority: Minor > Labels: jdk > Fix For: v2.6.0 > > > Class#newInstance is deprecated starting in Java 9 - > https://bugs.openjdk.java.net/browse/JDK-6850612 - because it may throw > undeclared checked exceptions. > The suggested replacement is getDeclaredConstructor().newInstance(), which > wraps the checked exceptions in InvocationException. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3417) Consider replacing ReentrantReadWriteLock with StampedLock
[
https://issues.apache.org/jira/browse/KYLIN-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624443#comment-16624443
]
Ted Yu edited comment on KYLIN-3417 at 10/1/18 4:24 PM:
For phase I, we don't need to use Optimistic read lock.
was (Author: yuzhih...@gmail.com):
For phase I, we don't need to use Optimistic read lock
> Consider replacing ReentrantReadWriteLock with StampedLock
> --
>
> Key: KYLIN-3417
> URL: https://issues.apache.org/jira/browse/KYLIN-3417
> Project: Kylin
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: jiatao.tao
>Priority: Major
> Fix For: Backlog
>
>
> ReentrantReadWriteLock's are only the right solution when there is long hold
> time due to expensive I/O.
> It is expensive for readers.
> We should see if the lighter {{StampedLock}} can be used instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-2650) Update to Apache Calcite Avatica 1.17.0
[ https://issues.apache.org/jira/browse/KYLIN-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-2650: -- Summary: Update to Apache Calcite Avatica 1.17.0 (was: Update to Apache Calcite Avatica 1.12.0) > Update to Apache Calcite Avatica 1.17.0 > --- > > Key: KYLIN-2650 > URL: https://issues.apache.org/jira/browse/KYLIN-2650 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Minor > > Apache Calcite Avatica 1.17.0 was released mid-July > https://sematext.com/opensee/m/Calcite/FR3K9IxYty1a4ECo1?subj=+ANNOUNCE+Apache+Calcite+Avatica+1+12+0+released > This issue upgrades Avatica dependency. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-2650) Update to Apache Calcite Avatica 1.12.0
[ https://issues.apache.org/jira/browse/KYLIN-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-2650: -- Description: Apache Calcite Avatica 1.17.0 was released mid-July https://sematext.com/opensee/m/Calcite/FR3K9IxYty1a4ECo1?subj=+ANNOUNCE+Apache+Calcite+Avatica+1+12+0+released This issue upgrades Avatica dependency. was: Apache Calcite Avatica 1.12.0 has just been released. https://sematext.com/opensee/m/Calcite/FR3K9IxYty1a4ECo1?subj=+ANNOUNCE+Apache+Calcite+Avatica+1+12+0+released This issue upgrades Avatica dependency. > Update to Apache Calcite Avatica 1.12.0 > --- > > Key: KYLIN-2650 > URL: https://issues.apache.org/jira/browse/KYLIN-2650 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Minor > > Apache Calcite Avatica 1.17.0 was released mid-July > https://sematext.com/opensee/m/Calcite/FR3K9IxYty1a4ECo1?subj=+ANNOUNCE+Apache+Calcite+Avatica+1+12+0+released > This issue upgrades Avatica dependency. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3310) Use lint for maven-compiler-plugin
[
https://issues.apache.org/jira/browse/KYLIN-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560940#comment-16560940
]
Ted Yu edited comment on KYLIN-3310 at 10/1/18 1:40 AM:
Thanks, Jiatao.
was (Author: yuzhih...@gmail.com):
Thanks, Jiatao .
> Use lint for maven-compiler-plugin
> --
>
> Key: KYLIN-3310
> URL: https://issues.apache.org/jira/browse/KYLIN-3310
> Project: Kylin
> Issue Type: Improvement
> Components: Tools, Build and Test
>Reporter: Ted Yu
>Assignee: jiatao.tao
>Priority: Major
> Fix For: v2.6.0
>
>
> lint helps identify structural problems.
> We should enable lint for maven-compiler-plugin
> {code}
> maven-compiler-plugin
> ${maven-compiler-plugin.version}
>
> 1.8
> 1.8
>
> -Xlint:all
> ${compiler.error.flag}
>
> -Xlint:-options
>
> -Xlint:-cast
> -Xlint:-deprecation
> -Xlint:-processing
> -Xlint:-rawtypes
> -Xlint:-serial
> -Xlint:-try
> -Xlint:-unchecked
> -Xlint:-varargs
>
>
>
>
> true
>
> false
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3394) Prepare for Kafka 2.0
[
https://issues.apache.org/jira/browse/KYLIN-3394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated KYLIN-3394:
--
Description:
Kafka 2.0 has been released.
I got the following when compiling against Kafka 2.0.0:
{code}
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:266:
error: type mismatch;
[ERROR] found : Option[org.apache.kafka.common.Node]
[ERROR] required: org.apache.kafka.common.Node
[ERROR] getBrokerInfoFromCache(zkUtils, cachedBrokerInfo,
List(l)).head.getNode(listenerName)
[ERROR]
^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:273:
error: type mismatch;
[ERROR] found : Seq[Option[org.apache.kafka.common.Node]]
[ERROR] required: Seq[org.apache.kafka.common.Node]
[ERROR] replicaInfo = getBrokerInfoFromCache(zkUtils,
cachedBrokerInfo, replicas).map(_.getNode(listenerName))
[ERROR]
^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:274:
error: type mismatch;
[ERROR] found : Seq[Option[org.apache.kafka.common.Node]]
[ERROR] required: Seq[org.apache.kafka.common.Node]
[ERROR] isrInfo = getBrokerInfoFromCache(zkUtils, cachedBrokerInfo,
inSyncReplicas).map(_.getNode(listenerName))
[ERROR]
^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:476:
error: value getConsumersInGroup is not a member of kafka.utils.ZkUtils
[ERROR] zkUtils.getConsumersInGroup(group).nonEmpty
[ERROR] ^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:489:
error: not found: type ZKGroupDirs
[ERROR] val dir = new ZKGroupDirs(group)
[ERROR] ^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:507:
error: value getTopicsByConsumerGroup is not a member of kafka.utils.ZkUtils
[ERROR] val topics = zkUtils.getTopicsByConsumerGroup(group)
[ERROR] ^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:512:
error: not found: type ZKGroupTopicDirs
[ERROR] val dir = new ZKGroupTopicDirs(group, topic)
[ERROR] ^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:528:
error: value getAllConsumerGroupsForTopic is not a member of kafka.utils.ZkUtils
[ERROR] val groups = zkUtils.getAllConsumerGroupsForTopic(topic)
[ERROR] ^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:720:
error: value encode is not a member of object kafka.utils.Json
[ERROR] val content =
Json.encode(getConfigChangeZnodeData(sanitizedEntityPath))
[ERROR]^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:733:
error: value encode is not a member of object kafka.utils.Json
[ERROR] zkUtils.updatePersistentPath(entityPath, Json.encode(map))
{code}
was:
Kafka 2.0 is around the corner.
I got the following when compiling against Kafka 2.0.0:
{code}
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:266:
error: type mismatch;
[ERROR] found : Option[org.apache.kafka.common.Node]
[ERROR] required: org.apache.kafka.common.Node
[ERROR] getBrokerInfoFromCache(zkUtils, cachedBrokerInfo,
List(l)).head.getNode(listenerName)
[ERROR]
^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:273:
error: type mismatch;
[ERROR] found : Seq[Option[org.apache.kafka.common.Node]]
[ERROR] required: Seq[org.apache.kafka.common.Node]
[ERROR] replicaInfo = getBrokerInfoFromCache(zkUtils,
cachedBrokerInfo, replicas).map(_.getNode(listenerName))
[ERROR]
^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:274:
error: type mismatch;
[ERROR] found : Seq[Option[org.apache.kafka.common.Node]]
[ERROR] required: Seq[org.apache.kafka.common.Node]
[ERROR] isrInfo = getBrokerInfoFromCache(zkUtils, cachedBrokerInfo,
inSyncReplicas).map(_.getNode(listenerName))
[ERROR]
^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:476:
error: value getConsumersInGroup is not a member of kafka.utils.ZkUtils
[ERROR] zkUtils.getConsumersInGroup(group).nonEmpty
[ERROR] ^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:489:
error: not found: type ZKGroupDirs
[ERROR] val dir = new ZKGroupDirs(group)
[ERROR] ^
[jira] [Updated] (KYLIN-3484) Update Hadoop version to 2.7.7
[ https://issues.apache.org/jira/browse/KYLIN-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3484: -- Description: We should upgrade the Hadoop 2.7 dependency to 2.7.7, to pick up bug and security fixes. (was: We should upgrade the Hadoop 2.7 dependency to 2.7.7, to pick up bug and security fixes .) > Update Hadoop version to 2.7.7 > -- > > Key: KYLIN-3484 > URL: https://issues.apache.org/jira/browse/KYLIN-3484 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Priority: Minor > > We should upgrade the Hadoop 2.7 dependency to 2.7.7, to pick up bug and > security fixes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3450) Consider using google re2j
[ https://issues.apache.org/jira/browse/KYLIN-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556761#comment-16556761 ] Ted Yu edited comment on KYLIN-3450 at 10/1/18 1:39 AM: It would be nice to see improvement from using re2j. was (Author: yuzhih...@gmail.com): It would be nice to see improvement from using re2j . > Consider using google re2j > -- > > Key: KYLIN-3450 > URL: https://issues.apache.org/jira/browse/KYLIN-3450 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Minor > > RE2J : https://github.com/google/re2j > For regular expression patterns with a high degree of alternation, using RE2J > would exhibit higher performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21261) Add log4j.properties for hbase-rsgroup tests
Ted Yu created HBASE-21261: -- Summary: Add log4j.properties for hbase-rsgroup tests Key: HBASE-21261 URL: https://issues.apache.org/jira/browse/HBASE-21261 Project: HBase Issue Type: Test Reporter: Ted Yu When I tried to debug TestRSGroups, at first I couldn't find any DEBUG log. Turns out that under hbase-rsgroup/src/test/resources there is no log4j.properties This issue adds log4j.properties for hbase-rsgroup tests. This would be useful when finding root cause for hbase-rsgroup test failure(s). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21261) Add log4j.properties for hbase-rsgroup tests
Ted Yu created HBASE-21261: -- Summary: Add log4j.properties for hbase-rsgroup tests Key: HBASE-21261 URL: https://issues.apache.org/jira/browse/HBASE-21261 Project: HBase Issue Type: Test Reporter: Ted Yu When I tried to debug TestRSGroups, at first I couldn't find any DEBUG log. Turns out that under hbase-rsgroup/src/test/resources there is no log4j.properties This issue adds log4j.properties for hbase-rsgroup tests. This would be useful when finding root cause for hbase-rsgroup test failure(s). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21258: --- Attachment: 21258.branch-2.v1.patch > Add resetting of flags for RS Group pre/post hooks in TestRSGroups > -- > > Key: HBASE-21258 > URL: https://issues.apache.org/jira/browse/HBASE-21258 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 21258.branch-1.04.txt, 21258.branch-1.05.txt, > 21258.branch-2.v1.patch, 21258.v1.txt > > > Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS > Group pre/post hooks in TestRSGroups was absent. > This issue is to add the resetting of these flags before each subtest starts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21207) Add client side sorting functionality in master web UI for table and region server details.
[ https://issues.apache.org/jira/browse/HBASE-21207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21207: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Add client side sorting functionality in master web UI for table and region > server details. > --- > > Key: HBASE-21207 > URL: https://issues.apache.org/jira/browse/HBASE-21207 > Project: HBase > Issue Type: Improvement > Components: master, monitoring, UI, Usability >Reporter: Archana Katiyar >Assignee: Archana Katiyar >Priority: Minor > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8 > > Attachments: 14926e82-b929-11e8-8bdd-4ce4621f1118.png, > 21207.branch-1.addendum.patch, 2724afd8-b929-11e8-8171-8b5b2ba3084e.png, > HBASE-21207-branch-1.patch, HBASE-21207-branch-1.v1.patch, > HBASE-21207-branch-2.v1.patch, HBASE-21207.patch, HBASE-21207.patch, > HBASE-21207.v1.patch, edc5c812-b928-11e8-87e2-ce6396629bbc.png > > > In Master UI, we can see region server details like requests per seconds and > number of regions etc. Similarly, for tables also we can see online regions , > offline regions. > It will help ops people in determining hot spotting if we can provide sort > functionality in the UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10468) Potential missing break for PARTITION_CUSTOM in OutputEmitter ctor
[
https://issues.apache.org/jira/browse/FLINK-10468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633416#comment-16633416
]
Ted Yu commented on FLINK-10468:
That was why I started the issue title with Potential.
Even if this is the case, assigning {{channels}} and breaking would make the
code easier to understand for other people.
Or, a comment should be added stating the fact.
> Potential missing break for PARTITION_CUSTOM in OutputEmitter ctor
> --
>
> Key: FLINK-10468
> URL: https://issues.apache.org/jira/browse/FLINK-10468
> Project: Flink
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>
> Here is related code:
> {code}
> switch (strategy) {
> case PARTITION_CUSTOM:
> extractedKeys = new Object[1];
> case FORWARD:
> {code}
> It seems a 'break' is missing prior to FORWARD case.
> {code}
> if (strategy == ShipStrategyType.PARTITION_CUSTOM && partitioner == null)
> {
> throw new NullPointerException("Partitioner must not be null when the
> ship strategy is set to custom partitioning.");
> }
> {code}
> Since the above check is for PARTITION_CUSTOM, it seems we can place the
> check in the switch statement.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-10468) Potential missing break for PARTITION_CUSTOM in OutputEmitter ctor
Ted Yu created FLINK-10468:
--
Summary: Potential missing break for PARTITION_CUSTOM in
OutputEmitter ctor
Key: FLINK-10468
URL: https://issues.apache.org/jira/browse/FLINK-10468
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
switch (strategy) {
case PARTITION_CUSTOM:
extractedKeys = new Object[1];
case FORWARD:
{code}
It seems a 'break' is missing prior to FORWARD case.
{code}
if (strategy == ShipStrategyType.PARTITION_CUSTOM && partitioner == null) {
throw new NullPointerException("Partitioner must not be null when the
ship strategy is set to custom partitioning.");
}
{code}
Since the above check is for PARTITION_CUSTOM, it seems we can place the check
in the switch statement.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-10468) Potential missing break for PARTITION_CUSTOM in OutputEmitter ctor
Ted Yu created FLINK-10468:
--
Summary: Potential missing break for PARTITION_CUSTOM in
OutputEmitter ctor
Key: FLINK-10468
URL: https://issues.apache.org/jira/browse/FLINK-10468
Project: Flink
Issue Type: Bug
Reporter: Ted Yu
Here is related code:
{code}
switch (strategy) {
case PARTITION_CUSTOM:
extractedKeys = new Object[1];
case FORWARD:
{code}
It seems a 'break' is missing prior to FORWARD case.
{code}
if (strategy == ShipStrategyType.PARTITION_CUSTOM && partitioner == null) {
throw new NullPointerException("Partitioner must not be null when the
ship strategy is set to custom partitioning.");
}
{code}
Since the above check is for PARTITION_CUSTOM, it seems we can place the check
in the switch statement.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3561) Upgrade to JUnit 5
[ https://issues.apache.org/jira/browse/KYLIN-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3561: -- Description: JUnit 5 brings multiple useful features so tests are easier to read and write. We can bump up the dependency version and create new tests with JUnit 5 features. Relevant features of JUnit 5: dynamic test, nested tests, parameterized tests https://twitter.com/nipafx/status/1027095088059559936 was: JUnit 5 brings multiple useful features so tests are easier to read and write. We can bump up the dependency version and create new tests with JUnit 5 features. Relevant features of JUnit 5: dynamic test, nested tests, parameterized tests https://twitter.com/nipafx/status/1027095088059559936 > Upgrade to JUnit 5 > -- > > Key: KYLIN-3561 > URL: https://issues.apache.org/jira/browse/KYLIN-3561 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Priority: Major > > JUnit 5 brings multiple useful features so tests are easier to read and write. > We can bump up the dependency version and create new tests with JUnit 5 > features. > Relevant features of JUnit 5: dynamic test, nested tests, parameterized tests > https://twitter.com/nipafx/status/1027095088059559936 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21207) Add client side sorting functionality in master web UI for table and region server details.
[
https://issues.apache.org/jira/browse/HBASE-21207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633243#comment-16633243
]
Ted Yu commented on HBASE-21207:
This is the command I used locally to verify the addendum:
{code}
mvn clean verify -fae --batch-mode -pl
hbase-shaded/hbase-shaded-check-invariants -am -Dtest=NoUnitTests
-DHBasePatchProcess -Prelease -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true
-Dfindbugs.skip=true
{code}
>From
>https://builds.apache.org/job/PreCommit-HBASE-Build/14548/artifact/patchprocess/archiver/hbase-server/target/rat.txt
> , I don't see file without license.
> Add client side sorting functionality in master web UI for table and region
> server details.
> ---
>
> Key: HBASE-21207
> URL: https://issues.apache.org/jira/browse/HBASE-21207
> Project: HBase
> Issue Type: Improvement
> Components: master, monitoring, UI, Usability
>Reporter: Archana Katiyar
>Assignee: Archana Katiyar
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8
>
> Attachments: 14926e82-b929-11e8-8bdd-4ce4621f1118.png,
> 21207.branch-1.addendum.patch, 2724afd8-b929-11e8-8171-8b5b2ba3084e.png,
> HBASE-21207-branch-1.patch, HBASE-21207-branch-1.v1.patch,
> HBASE-21207-branch-2.v1.patch, HBASE-21207.patch, HBASE-21207.patch,
> HBASE-21207.v1.patch, edc5c812-b928-11e8-87e2-ce6396629bbc.png
>
>
> In Master UI, we can see region server details like requests per seconds and
> number of regions etc. Similarly, for tables also we can see online regions ,
> offline regions.
> It will help ops people in determining hot spotting if we can provide sort
> functionality in the UI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-10467) Upgrade commons-compress to 1.18
Ted Yu created FLINK-10467: -- Summary: Upgrade commons-compress to 1.18 Key: FLINK-10467 URL: https://issues.apache.org/jira/browse/FLINK-10467 Project: Flink Issue Type: Task Reporter: Ted Yu org.apache.commons:commons-compress defines an API for working with compression and archive formats. Affected versions of this package are vulnerable to Directory Traversal. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-10467) Upgrade commons-compress to 1.18
Ted Yu created FLINK-10467: -- Summary: Upgrade commons-compress to 1.18 Key: FLINK-10467 URL: https://issues.apache.org/jira/browse/FLINK-10467 Project: Flink Issue Type: Task Reporter: Ted Yu org.apache.commons:commons-compress defines an API for working with compression and archive formats. Affected versions of this package are vulnerable to Directory Traversal. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21207) Add client side sorting functionality in master web UI for table and region server details.
[ https://issues.apache.org/jira/browse/HBASE-21207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21207: --- Status: Patch Available (was: Reopened) Corrected the exclusion in pom.xml for branch-1. Master branch doesn't have this problem. > Add client side sorting functionality in master web UI for table and region > server details. > --- > > Key: HBASE-21207 > URL: https://issues.apache.org/jira/browse/HBASE-21207 > Project: HBase > Issue Type: Improvement > Components: master, monitoring, UI, Usability >Reporter: Archana Katiyar >Assignee: Archana Katiyar >Priority: Minor > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8 > > Attachments: 14926e82-b929-11e8-8bdd-4ce4621f1118.png, > 21207.branch-1.addendum.patch, 2724afd8-b929-11e8-8171-8b5b2ba3084e.png, > HBASE-21207-branch-1.patch, HBASE-21207-branch-1.v1.patch, > HBASE-21207-branch-2.v1.patch, HBASE-21207.patch, HBASE-21207.patch, > HBASE-21207.v1.patch, edc5c812-b928-11e8-87e2-ce6396629bbc.png > > > In Master UI, we can see region server details like requests per seconds and > number of regions etc. Similarly, for tables also we can see online regions , > offline regions. > It will help ops people in determining hot spotting if we can provide sort > functionality in the UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21207) Add client side sorting functionality in master web UI for table and region server details.
[ https://issues.apache.org/jira/browse/HBASE-21207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21207: --- Attachment: 21207.branch-1.addendum.patch > Add client side sorting functionality in master web UI for table and region > server details. > --- > > Key: HBASE-21207 > URL: https://issues.apache.org/jira/browse/HBASE-21207 > Project: HBase > Issue Type: Improvement > Components: master, monitoring, UI, Usability >Reporter: Archana Katiyar >Assignee: Archana Katiyar >Priority: Minor > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8 > > Attachments: 14926e82-b929-11e8-8bdd-4ce4621f1118.png, > 21207.branch-1.addendum.patch, 2724afd8-b929-11e8-8171-8b5b2ba3084e.png, > HBASE-21207-branch-1.patch, HBASE-21207-branch-1.v1.patch, > HBASE-21207-branch-2.v1.patch, HBASE-21207.patch, HBASE-21207.patch, > HBASE-21207.v1.patch, edc5c812-b928-11e8-87e2-ce6396629bbc.png > > > In Master UI, we can see region server details like requests per seconds and > number of regions etc. Similarly, for tables also we can see online regions , > offline regions. > It will help ops people in determining hot spotting if we can provide sort > functionality in the UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (HBASE-21207) Add client side sorting functionality in master web UI for table and region server details.
[ https://issues.apache.org/jira/browse/HBASE-21207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reopened HBASE-21207: > Add client side sorting functionality in master web UI for table and region > server details. > --- > > Key: HBASE-21207 > URL: https://issues.apache.org/jira/browse/HBASE-21207 > Project: HBase > Issue Type: Improvement > Components: master, monitoring, UI, Usability >Reporter: Archana Katiyar >Assignee: Archana Katiyar >Priority: Minor > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8 > > Attachments: 14926e82-b929-11e8-8bdd-4ce4621f1118.png, > 21207.branch-1.addendum.patch, 2724afd8-b929-11e8-8171-8b5b2ba3084e.png, > HBASE-21207-branch-1.patch, HBASE-21207-branch-1.v1.patch, > HBASE-21207-branch-2.v1.patch, HBASE-21207.patch, HBASE-21207.patch, > HBASE-21207.v1.patch, edc5c812-b928-11e8-87e2-ce6396629bbc.png > > > In Master UI, we can see region server details like requests per seconds and > number of regions etc. Similarly, for tables also we can see online regions , > offline regions. > It will help ops people in determining hot spotting if we can provide sort > functionality in the UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (HBASE-21207) Add client side sorting functionality in master web UI for table and region server details.
[ https://issues.apache.org/jira/browse/HBASE-21207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reopened HBASE-21207: > Add client side sorting functionality in master web UI for table and region > server details. > --- > > Key: HBASE-21207 > URL: https://issues.apache.org/jira/browse/HBASE-21207 > Project: HBase > Issue Type: Improvement > Components: master, monitoring, UI, Usability >Reporter: Archana Katiyar >Assignee: Archana Katiyar >Priority: Minor > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8 > > Attachments: 14926e82-b929-11e8-8bdd-4ce4621f1118.png, > 21207.branch-1.addendum.patch, 2724afd8-b929-11e8-8171-8b5b2ba3084e.png, > HBASE-21207-branch-1.patch, HBASE-21207-branch-1.v1.patch, > HBASE-21207-branch-2.v1.patch, HBASE-21207.patch, HBASE-21207.patch, > HBASE-21207.v1.patch, edc5c812-b928-11e8-87e2-ce6396629bbc.png > > > In Master UI, we can see region server details like requests per seconds and > number of regions etc. Similarly, for tables also we can see online regions , > offline regions. > It will help ops people in determining hot spotting if we can provide sort > functionality in the UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-2690) Upgrade calcite to release 1.18
[ https://issues.apache.org/jira/browse/KYLIN-2690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-2690: -- Summary: Upgrade calcite to release 1.18 (was: Upgrade calcite to release 1.17) > Upgrade calcite to release 1.18 > --- > > Key: KYLIN-2690 > URL: https://issues.apache.org/jira/browse/KYLIN-2690 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Major > > Calcite 1.16 has been released. > We should consider upgrading to this release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-2690) Upgrade calcite to release 1.18
[ https://issues.apache.org/jira/browse/KYLIN-2690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-2690: -- Description: Calcite 1.18 has been released. We should consider upgrading to this release. was: Calcite 1.16 has been released. We should consider upgrading to this release. > Upgrade calcite to release 1.18 > --- > > Key: KYLIN-2690 > URL: https://issues.apache.org/jira/browse/KYLIN-2690 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Major > > Calcite 1.18 has been released. > We should consider upgrading to this release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3272) Upgrade Spark dependency to 2.3.2
[ https://issues.apache.org/jira/browse/KYLIN-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3272: -- Description: Currently Spark 2.1.2 is used. Spark 2.3.2 was released. We should upgrade the dependency to 2.3.2 https://sematext.com/opensee/m/Spark/q3RTtb4Q9LI4F6F?subj=+ANNOUNCE+Announcing+Apache+Spark+2+3+2 was: Currently Spark 2.1.2 is used. Spark 2.3.1 was released. We should upgrade the dependency to 2.3.1 > Upgrade Spark dependency to 2.3.2 > - > > Key: KYLIN-3272 > URL: https://issues.apache.org/jira/browse/KYLIN-3272 > Project: Kylin > Issue Type: Improvement > Components: Spark Engine >Reporter: Ted Yu >Priority: Minor > > Currently Spark 2.1.2 is used. > Spark 2.3.2 was released. > We should upgrade the dependency to 2.3.2 > https://sematext.com/opensee/m/Spark/q3RTtb4Q9LI4F6F?subj=+ANNOUNCE+Announcing+Apache+Spark+2+3+2 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21207) Add client side sorting functionality in master web UI for table and region server details.
[
https://issues.apache.org/jira/browse/HBASE-21207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633172#comment-16633172
]
Ted Yu commented on HBASE-21207:
hbase-server/src/main/resources/hbase-webapps/static/js/jquery.tablesorter.min.js
is causing rat check failure.
{code}
/*! jQuery jquery.com | jquery.org/license */
{code}
What can we do about the above ?
> Add client side sorting functionality in master web UI for table and region
> server details.
> ---
>
> Key: HBASE-21207
> URL: https://issues.apache.org/jira/browse/HBASE-21207
> Project: HBase
> Issue Type: Improvement
> Components: master, monitoring, UI, Usability
>Reporter: Archana Katiyar
>Assignee: Archana Katiyar
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8
>
> Attachments: 14926e82-b929-11e8-8bdd-4ce4621f1118.png,
> 2724afd8-b929-11e8-8171-8b5b2ba3084e.png, HBASE-21207-branch-1.patch,
> HBASE-21207-branch-1.v1.patch, HBASE-21207-branch-2.v1.patch,
> HBASE-21207.patch, HBASE-21207.patch, HBASE-21207.v1.patch,
> edc5c812-b928-11e8-87e2-ce6396629bbc.png
>
>
> In Master UI, we can see region server details like requests per seconds and
> number of regions etc. Similarly, for tables also we can see online regions ,
> offline regions.
> It will help ops people in determining hot spotting if we can provide sort
> functionality in the UI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3519) Upgrade Jacoco version to 0.8.2
[ https://issues.apache.org/jira/browse/KYLIN-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3519: -- Description: Jacoco 0.8.2 adds Java 11 support: https://github.com/jacoco/jacoco/releases/tag/v0.8.2 Java 11 RC1 is out. We should consider upgrading Jacoco. was: Jacoco 0.8.2 adds Java 11 support: https://github.com/jacoco/jacoco/releases/tag/v0.8.2 Java 11 RC1 is out. We should consider upgrading Jacoco. > Upgrade Jacoco version to 0.8.2 > --- > > Key: KYLIN-3519 > URL: https://issues.apache.org/jira/browse/KYLIN-3519 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Minor > > Jacoco 0.8.2 adds Java 11 support: >https://github.com/jacoco/jacoco/releases/tag/v0.8.2 > Java 11 RC1 is out. > We should consider upgrading Jacoco. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3572) Upgrade commons-dbcp to latest commons-dbcp2
[ https://issues.apache.org/jira/browse/KYLIN-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3572: -- Description: There have been many improvements and a major release in commons-dbcp. The latest version is 2.5.0. This Jira proposes to upgrade the version of commons-dbcp to commons-dbcp2 version 2.5.0 in order to benefit from the improvements/additions. This will involve some code changes in addition to the updated dependencies, as the API has changed slightly to use different terminology was: There have been many improvements and a major release in commons-dbcp. The latest version is 2.5.0. This Jira proposes to upgrade the version of commons-dbcp to commons-dbcp2 version 2.5.0 in order to benefit from the improvements/additions. This will involve some code changes in addition to the updated dependencies, as the API has changed slightly to use different terminology > Upgrade commons-dbcp to latest commons-dbcp2 > > > Key: KYLIN-3572 > URL: https://issues.apache.org/jira/browse/KYLIN-3572 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Priority: Major > > There have been many improvements and a major release in commons-dbcp. The > latest version is 2.5.0. > This Jira proposes to upgrade the version of commons-dbcp to commons-dbcp2 > version 2.5.0 in order to benefit from the improvements/additions. This will > involve some code changes in addition to the updated dependencies, as the API > has changed slightly to use different terminology -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3573) Upgrade Apache parent POM to version 21
[ https://issues.apache.org/jira/browse/KYLIN-3573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3573: -- Description: Apache parent POM 21 has just been released. It generates .sha512 checksums during release with Maven Release Plugin. We should consider upgrading parent POM. was: Apache parent POM 21 has just been released. It generates .sha512 checksums during release with Maven Release Plugin. We should consider upgrading parent POM. > Upgrade Apache parent POM to version 21 > --- > > Key: KYLIN-3573 > URL: https://issues.apache.org/jira/browse/KYLIN-3573 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Priority: Major > > Apache parent POM 21 has just been released. It generates .sha512 checksums > during release with Maven Release Plugin. > We should consider upgrading parent POM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3523) Upgrade hadoop and hbase dependencies in master-hadoop3.1 branch
[ https://issues.apache.org/jira/browse/KYLIN-3523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3523: -- Description: Hadoop 3.1.1 has been released as stable. hbase 2.0.2 vote has passed. We should upgrade to these two versions in master-hadoop3.1 branch. was: Hadoop 3.1.1 has been released as stable. hbase 2.0.2 vote seems to pass. We should upgrade to these two versions in master-hadoop3.1 branch. > Upgrade hadoop and hbase dependencies in master-hadoop3.1 branch > > > Key: KYLIN-3523 > URL: https://issues.apache.org/jira/browse/KYLIN-3523 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Major > > Hadoop 3.1.1 has been released as stable. > hbase 2.0.2 vote has passed. > We should upgrade to these two versions in master-hadoop3.1 branch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[
https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633161#comment-16633161
]
Ted Yu commented on HBASE-21258:
For patch v5, the following shows that only two subtests run:
{code}
grep 'server(244): calling resetFlag'
hbase-rsgroup/target/surefire-reports/org.apache.hadoop.hbase.rsgroup.TestRSGroups1-output.txt
{code}
> Add resetting of flags for RS Group pre/post hooks in TestRSGroups
> --
>
> Key: HBASE-21258
> URL: https://issues.apache.org/jira/browse/HBASE-21258
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 21258.branch-1.04.txt, 21258.branch-1.05.txt,
> 21258.v1.txt
>
>
> Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS
> Group pre/post hooks in TestRSGroups was absent.
> This issue is to add the resetting of these flags before each subtest starts.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21258: --- Attachment: 21258.branch-1.05.txt > Add resetting of flags for RS Group pre/post hooks in TestRSGroups > -- > > Key: HBASE-21258 > URL: https://issues.apache.org/jira/browse/HBASE-21258 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 21258.branch-1.04.txt, 21258.branch-1.05.txt, > 21258.v1.txt > > > Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS > Group pre/post hooks in TestRSGroups was absent. > This issue is to add the resetting of these flags before each subtest starts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21258: --- Attachment: 21258.branch-1.04.txt > Add resetting of flags for RS Group pre/post hooks in TestRSGroups > -- > > Key: HBASE-21258 > URL: https://issues.apache.org/jira/browse/HBASE-21258 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 21258.branch-1.04.txt, 21258.v1.txt > > > Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS > Group pre/post hooks in TestRSGroups was absent. > This issue is to add the resetting of these flags before each subtest starts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[
https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633139#comment-16633139
]
Ted Yu commented on HBASE-21258:
Even for a run where testNamespaceConstraint failed, we can see that
preAddRSGroupCalled was set (additional log):
{code}
2018-09-29 22:33:39,357 DEBUG [main]
rsgroup.TestRSGroups$CPMasterObserver(288): calling resetFlags
2018-09-29 22:33:39,357 INFO [main] rsgroup.TestRSGroups(221):
testNamespaceConstraint
2018-09-29 22:33:39,358 DEBUG
[RpcServer.default.FPBQ.Fifo.handler=27,queue=0,port=39540]
rsgroup.TestRSGroups$CPMasterObserver(340): setting preAddRSGroupCalled
{code}
> Add resetting of flags for RS Group pre/post hooks in TestRSGroups
> --
>
> Key: HBASE-21258
> URL: https://issues.apache.org/jira/browse/HBASE-21258
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 21258.v1.txt
>
>
> Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS
> Group pre/post hooks in TestRSGroups was absent.
> This issue is to add the resetting of these flags before each subtest starts.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633136#comment-16633136 ] Ted Yu commented on HBASE-21258: Interesting - I couldn't reproduce test failure on branch-1 when running single subtest. Each complete run of the test takes 280 seconds, making debugging prolonged. > Add resetting of flags for RS Group pre/post hooks in TestRSGroups > -- > > Key: HBASE-21258 > URL: https://issues.apache.org/jira/browse/HBASE-21258 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 21258.v1.txt > > > Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS > Group pre/post hooks in TestRSGroups was absent. > This issue is to add the resetting of these flags before each subtest starts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633125#comment-16633125 ] Ted Yu commented on HBASE-21258: I ran the 7 tests in TestRSGroups by commenting out the tests in TestRSGroupsBase. They all passed with branch-1 patch. > Add resetting of flags for RS Group pre/post hooks in TestRSGroups > -- > > Key: HBASE-21258 > URL: https://issues.apache.org/jira/browse/HBASE-21258 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 21258.v1.txt > > > Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS > Group pre/post hooks in TestRSGroups was absent. > This issue is to add the resetting of these flags before each subtest starts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21258: --- Attachment: (was: 21258.v1.txt) > Add resetting of flags for RS Group pre/post hooks in TestRSGroups > -- > > Key: HBASE-21258 > URL: https://issues.apache.org/jira/browse/HBASE-21258 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 21258.v1.txt > > > Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS > Group pre/post hooks in TestRSGroups was absent. > This issue is to add the resetting of these flags before each subtest starts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21258: --- Attachment: 21258.v1.txt > Add resetting of flags for RS Group pre/post hooks in TestRSGroups > -- > > Key: HBASE-21258 > URL: https://issues.apache.org/jira/browse/HBASE-21258 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 21258.v1.txt > > > Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS > Group pre/post hooks in TestRSGroups was absent. > This issue is to add the resetting of these flags before each subtest starts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21198) Exclude dependency on net.minidev:json-smart
[
https://issues.apache.org/jira/browse/HBASE-21198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633106#comment-16633106
]
Ted Yu commented on HBASE-21198:
bq. Where did it come in?
json-smart comes in as transitive dependency:
{code}
[INFO] | \- org.apache.hadoop:hadoop-auth:jar:3.0.0:compile
[INFO] | +- com.nimbusds:nimbus-jose-jwt:jar:4.41.1:compile
[INFO] | | +- com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile
[INFO] | | \- net.minidev:json-smart:jar:2.3:compile (version selected
from constraint [1.3.1,2.3])
{code}
bq. Why did it come in?
For hadoop-2.0 profile in hbase-client/pom.xml , we have:
{code}
org.apache.hadoop
hadoop-auth
{code}
The transitive dependency on json-smart seems not needed.
bq. Was it a mistake?
If you meant introducing the dependency transitively was a mistake, I would
agree.
> Exclude dependency on net.minidev:json-smart
>
>
> Key: HBASE-21198
> URL: https://issues.apache.org/jira/browse/HBASE-21198
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Priority: Major
>
> From
> https://builds.apache.org/job/PreCommit-HBASE-Build/14414/artifact/patchprocess/patch-javac-3.0.0.txt
> :
> {code}
> [ERROR] Failed to execute goal on project hbase-common: Could not resolve
> dependencies for project org.apache.hbase:hbase-common:jar:3.0.0-SNAPSHOT:
> Failed to collect dependencies at org.apache.hadoop:hadoop-common:jar:3.0.0
> -> org.apache.hadoop:hadoop-auth:jar:3.0.0 ->
> com.nimbusds:nimbus-jose-jwt:jar:4.41.1 ->
> net.minidev:json-smart:jar:2.3-SNAPSHOT: Failed to read artifact descriptor
> for net.minidev:json-smart:jar:2.3-SNAPSHOT: Could not transfer artifact
> net.minidev:json-smart:pom:2.3-SNAPSHOT from/to dynamodb-local-oregon
> (https://s3-us-west-2.amazonaws.com/dynamodb-local/release): Access denied
> to:
> https://s3-us-west-2.amazonaws.com/dynamodb-local/release/net/minidev/json-smart/2.3-SNAPSHOT/json-smart-2.3-SNAPSHOT.pom
> , ReasonPhrase:Forbidden. -> [Help 1]
> {code}
> We should exclude dependency on net.minidev:json-smart
> hbase-common/bin/pom.xml has done so.
> The other pom.xml should do the same.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21175) Partially initialized SnapshotHFileCleaner leads to NPE during TestHFileArchiving
[
https://issues.apache.org/jira/browse/HBASE-21175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-21175:
---
Labels: snapshot (was: )
> Partially initialized SnapshotHFileCleaner leads to NPE during
> TestHFileArchiving
> -
>
> Key: HBASE-21175
> URL: https://issues.apache.org/jira/browse/HBASE-21175
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Priority: Minor
> Labels: snapshot
>
> TestHFileArchiving#testCleaningRace creates HFileCleaner instance within the
> test.
> When SnapshotHFileCleaner.init() is called, there is no master parameter
> passed in {{params}}.
> When the chore runs the cleaner during the test, NPE comes out of this line
> in getDeletableFiles():
> {code}
> return cache.getUnreferencedFiles(files, master.getSnapshotManager());
> {code}
> since master is null.
> We should either check for the null master or, pass master instance properly
> when constructing the cleaner instance.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21149) TestIncrementalBackupWithBulkLoad may fail due to file copy failure
[
https://issues.apache.org/jira/browse/HBASE-21149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-21149:
---
Status: Open (was: Patch Available)
> TestIncrementalBackupWithBulkLoad may fail due to file copy failure
> ---
>
> Key: HBASE-21149
> URL: https://issues.apache.org/jira/browse/HBASE-21149
> Project: HBase
> Issue Type: Test
> Components: backuprestore
>Reporter: Ted Yu
>Assignee: Vladimir Rodionov
>Priority: Major
> Attachments: HBASE-21149-v1.patch
>
>
> From
> https://builds.apache.org/job/HBase%20Nightly/job/master/471/testReport/junit/org.apache.hadoop.hbase.backup/TestIncrementalBackupWithBulkLoad/TestIncBackupDeleteTable/
> :
> {code}
> 2018-09-03 11:54:30,526 ERROR [Time-limited test]
> impl.TableBackupClient(235): Unexpected Exception : Failed copy from
> hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/0f626c66493649daaf84057b8dd71a30_SeqId_205_,hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/ad8df6415bd9459d9b3df76c588d79df_SeqId_205_
> to hdfs://localhost:53075/backupUT/backup_1535975655488
> java.io.IOException: Failed copy from
> hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/0f626c66493649daaf84057b8dd71a30_SeqId_205_,hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/ad8df6415bd9459d9b3df76c588d79df_SeqId_205_
> to hdfs://localhost:53075/backupUT/backup_1535975655488
> at
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.incrementalCopyHFiles(IncrementalTableBackupClient.java:351)
> at
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.copyBulkLoadedFiles(IncrementalTableBackupClient.java:219)
> at
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.handleBulkLoad(IncrementalTableBackupClient.java:198)
> at
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.execute(IncrementalTableBackupClient.java:320)
> at
> org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.backupTables(BackupAdminImpl.java:605)
> at
> org.apache.hadoop.hbase.backup.TestIncrementalBackupWithBulkLoad.TestIncBackupDeleteTable(TestIncrementalBackupWithBulkLoad.java:104)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> {code}
> However, some part of the test output was lost:
> {code}
> 2018-09-03 11:53:36,793 DEBUG [RS:0;765c9ca5ea28:36357] regions
> ...[truncated 398396 chars]...
> 8)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[
https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632958#comment-16632958
]
Ted Yu commented on HBASE-21256:
Zephyr:
{code}
450 if (multipleUnevenColumnFamilies) {
451 int n =
context.getConfiguration().getInt(BIG_FAMILY_VALUE_SIZE_KEY, 256);
{code}
Since you added precondition checks in your patch, can you add one for the
above against the value of hbase.client.keyvalue.maxsize ?
thanks
> Improve IntegrationTestBigLinkedList for testing huge data
> --
>
> Key: HBASE-21256
> URL: https://issues.apache.org/jira/browse/HBASE-21256
> Project: HBase
> Issue Type: Improvement
> Components: integration tests
>Affects Versions: 3.0.0
>Reporter: Zephyr Guo
>Assignee: Zephyr Guo
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21256-v1.patch, ITBLL-1.png, ITBLL-2.png
>
>
> Recently, I use ITBLL to test some features in our company. I have
> encountered the following problems:
>
> 1. Generator is too slow at the generating stage, the root cause is
> SecureRandom. There is a global lock in SecureRandom( See the following
> picture). I use Random instead of SecureRandom, and it could speed up this
> stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but
> speaking of generating random bytes, in my opnion,
> it is the same with Random.
> !ITBLL-1.png!
> 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by
> create keyString variable. However, keyString may never be used if test
> result is correct.(and that's in most cases). Just delay creating keyString
> can yield huge performance boost in verifing stage.
> !ITBLL-2.png!
> 3.Arguments check is needed, because there's constraint between arguments. If
> we broken this constraint, we can not get a correct circular list.
>
> 4.Let big family value size could be configured.
>
> 5.Avoid starting RS at backup master
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20627) Relocate RS Group pre/post hooks from RSGroupAdminServer to RSGroupAdminEndpoint
[
https://issues.apache.org/jira/browse/HBASE-20627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632945#comment-16632945
]
Ted Yu edited comment on HBASE-20627 at 9/29/18 11:17 AM:
--
Thanks for the reminder - looks like I dropped some local edit in the patch
here.
Since this was resolved 4 months ago, I opened HBASE-21258 to add the
resetting. As you can see from the patch there, assertions of the respective
flag pass with resetting.
was (Author: yuzhih...@gmail.com):
Thanks for the reminder - looks like I dropped some local edit in the patch
here.
Since this has been resolved 3 months ago, I am opening a new issue to add the
resetting. As you can see from the patch there, assertions of the respective
flag pass with resetting.
> Relocate RS Group pre/post hooks from RSGroupAdminServer to
> RSGroupAdminEndpoint
>
>
> Key: HBASE-20627
> URL: https://issues.apache.org/jira/browse/HBASE-20627
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 2.1.0, 1.4.5
>
> Attachments: 20627.branch-1.txt, 20627.v1.txt, 20627.v2.txt,
> 20627.v3.txt
>
>
> Currently RS Group pre/post hooks are called from RSGroupAdminServer.
> e.g. RSGroupAdminServer#removeRSGroup :
> {code}
> if (master.getMasterCoprocessorHost() != null) {
> master.getMasterCoprocessorHost().preRemoveRSGroup(name);
> }
> {code}
> RSGroupAdminServer#removeRSGroup is called by RSGroupAdminEndpoint :
> {code}
> checkPermission("removeRSGroup");
> groupAdminServer.removeRSGroup(request.getRSGroupName());
> {code}
> If permission check fails, the pre hook wouldn't be called.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21258: --- Attachment: 21258.v1.txt > Add resetting of flags for RS Group pre/post hooks in TestRSGroups > -- > > Key: HBASE-21258 > URL: https://issues.apache.org/jira/browse/HBASE-21258 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 21258.v1.txt > > > Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS > Group pre/post hooks in TestRSGroups was absent. > This issue is to add the resetting of these flags before each subtest starts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21258: --- Status: Patch Available (was: Open) > Add resetting of flags for RS Group pre/post hooks in TestRSGroups > -- > > Key: HBASE-21258 > URL: https://issues.apache.org/jira/browse/HBASE-21258 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 21258.v1.txt > > > Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS > Group pre/post hooks in TestRSGroups was absent. > This issue is to add the resetting of these flags before each subtest starts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632950#comment-16632950 ] Ted Yu commented on HBASE-21258: [~xucang]: I ran the test locally with patch which passed. Can you take a look at the patch ? > Add resetting of flags for RS Group pre/post hooks in TestRSGroups > -- > > Key: HBASE-21258 > URL: https://issues.apache.org/jira/browse/HBASE-21258 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 21258.v1.txt > > > Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS > Group pre/post hooks in TestRSGroups was absent. > This issue is to add the resetting of these flags before each subtest starts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
Ted Yu created HBASE-21258: -- Summary: Add resetting of flags for RS Group pre/post hooks in TestRSGroups Key: HBASE-21258 URL: https://issues.apache.org/jira/browse/HBASE-21258 Project: HBase Issue Type: Test Reporter: Ted Yu Assignee: Ted Yu Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS Group pre/post hooks in TestRSGroups was absent. This issue is to add the resetting of these flags before each subtest starts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
Ted Yu created HBASE-21258: -- Summary: Add resetting of flags for RS Group pre/post hooks in TestRSGroups Key: HBASE-21258 URL: https://issues.apache.org/jira/browse/HBASE-21258 Project: HBase Issue Type: Test Reporter: Ted Yu Assignee: Ted Yu Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS Group pre/post hooks in TestRSGroups was absent. This issue is to add the resetting of these flags before each subtest starts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20627) Relocate RS Group pre/post hooks from RSGroupAdminServer to RSGroupAdminEndpoint
[
https://issues.apache.org/jira/browse/HBASE-20627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632945#comment-16632945
]
Ted Yu commented on HBASE-20627:
Thanks for the reminder - looks like I dropped some local edit in the patch
here.
Since this has been resolved 3 months ago, I am opening a new issue to add the
resetting. As you can see from the patch there, assertions of the respective
flag pass with resetting.
> Relocate RS Group pre/post hooks from RSGroupAdminServer to
> RSGroupAdminEndpoint
>
>
> Key: HBASE-20627
> URL: https://issues.apache.org/jira/browse/HBASE-20627
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 2.1.0, 1.4.5
>
> Attachments: 20627.branch-1.txt, 20627.v1.txt, 20627.v2.txt,
> 20627.v3.txt
>
>
> Currently RS Group pre/post hooks are called from RSGroupAdminServer.
> e.g. RSGroupAdminServer#removeRSGroup :
> {code}
> if (master.getMasterCoprocessorHost() != null) {
> master.getMasterCoprocessorHost().preRemoveRSGroup(name);
> }
> {code}
> RSGroupAdminServer#removeRSGroup is called by RSGroupAdminEndpoint :
> {code}
> checkPermission("removeRSGroup");
> groupAdminServer.removeRSGroup(request.getRSGroupName());
> {code}
> If permission check fails, the pre hook wouldn't be called.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21256: --- Status: Patch Available (was: Open) > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21256-v1.patch, ITBLL-1.png, ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMBARI-22965) Boolean#getBoolean is used to parse value
[
https://issues.apache.org/jira/browse/AMBARI-22965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated AMBARI-22965:
Description:
In StormTimelineMetricsReporter.java :
{code}
if (cf.containsKey(SET_INSTANCE_ID_PROPERTY)) {
setInstanceId =
Boolean.getBoolean(cf.get(SET_INSTANCE_ID_PROPERTY).toString());
{code}
According to
https://docs.oracle.com/javase/7/docs/api/java/lang/Boolean.html#getBoolean(java.lang.String):
{code}
Returns true if and only if the system property named by the argument exists
and is equal to the string "true"
{code}
This was not the intention of the quoted code.
FlumeTimelineMetricsSink.java may have the same issue.
was:
In StormTimelineMetricsReporter.java :
{code}
if (cf.containsKey(SET_INSTANCE_ID_PROPERTY)) {
setInstanceId =
Boolean.getBoolean(cf.get(SET_INSTANCE_ID_PROPERTY).toString());
{code}
According to
https://docs.oracle.com/javase/7/docs/api/java/lang/Boolean.html#getBoolean(java.lang.String):
{code}
Returns true if and only if the system property named by the argument exists
and is equal to the string "true"
{code}
This was not the intention of the quoted code.
FlumeTimelineMetricsSink.java may have the same issue.
> Boolean#getBoolean is used to parse value
> -
>
> Key: AMBARI-22965
> URL: https://issues.apache.org/jira/browse/AMBARI-22965
> Project: Ambari
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Major
>
> In StormTimelineMetricsReporter.java :
> {code}
> if (cf.containsKey(SET_INSTANCE_ID_PROPERTY)) {
> setInstanceId =
> Boolean.getBoolean(cf.get(SET_INSTANCE_ID_PROPERTY).toString());
> {code}
> According to
> https://docs.oracle.com/javase/7/docs/api/java/lang/Boolean.html#getBoolean(java.lang.String):
> {code}
> Returns true if and only if the system property named by the argument exists
> and is equal to the string "true"
> {code}
> This was not the intention of the quoted code.
> FlumeTimelineMetricsSink.java may have the same issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AMBARI-22621) Ensure value for hbase.coprocessor.abortonerror is true
[ https://issues.apache.org/jira/browse/AMBARI-22621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated AMBARI-22621: Description: In the coprocessor refactor for hbase-2, Server#abort has been taken out of reach. We should ensure that value for hbase.coprocessor.abortonerror is true so that coprocessor can abort the server by throwing exception. See HBASE-19341 for related details. was: In the coprocessor refactor for hbase-2, Server#abort has been taken out of reach. We should ensure that value for hbase.coprocessor.abortonerror is true so that coprocessor can abort the server by throwing exception. See HBASE-19341 for related details. > Ensure value for hbase.coprocessor.abortonerror is true > --- > > Key: AMBARI-22621 > URL: https://issues.apache.org/jira/browse/AMBARI-22621 > Project: Ambari > Issue Type: Improvement >Reporter: Ted Yu >Priority: Major > > In the coprocessor refactor for hbase-2, Server#abort has been taken out of > reach. > We should ensure that value for hbase.coprocessor.abortonerror is true so > that coprocessor can abort the server by throwing exception. > See HBASE-19341 for related details. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21194) Add TestCopyTable which exercises MOB feature
[ https://issues.apache.org/jira/browse/HBASE-21194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21194: --- Labels: mob (was: ) > Add TestCopyTable which exercises MOB feature > - > > Key: HBASE-21194 > URL: https://issues.apache.org/jira/browse/HBASE-21194 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Priority: Minor > Labels: mob > > Currently TestCopyTable doesn't cover table(s) with MOB feature enabled. > We should add variant that enables MOB on the table being copied and verify > that MOB content is copied correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21196) HTableMultiplexer clears the meta cache after every put operation
[
https://issues.apache.org/jira/browse/HBASE-21196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632658#comment-16632658
]
Ted Yu commented on HBASE-21196:
Thanks for the commit, Andrew.
> HTableMultiplexer clears the meta cache after every put operation
> -
>
> Key: HBASE-21196
> URL: https://issues.apache.org/jira/browse/HBASE-21196
> Project: HBase
> Issue Type: Bug
> Components: Performance
>Affects Versions: 3.0.0, 1.3.3, 2.2.0
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Critical
> Fix For: 3.0.0
>
> Attachments: HBASE-21196.master.001.patch,
> HBASE-21196.master.001.patch, HBASE-21196.master.002.patch,
> HTableMultiplexer1000Puts.UT.txt
>
>
> *Problem:* Operations which use
> {{AsyncRequestFutureImpl.receiveMultiAction(MultiAction, ServerName,
> MultiResponse, int)}} API with tablename set to null reset the meta cache of
> the corresponding server after each call. One such operation is put operation
> of HTableMultiplexer (Might not be the only one). This may impact the
> performance of the system severely as all new ops directed to that server
> will have to go to zk first to get the meta table address and then get the
> location of the table region as it will become empty after every
> htablemultiplexer put.
> From the logs below, one can see after every other put the cached region
> locations are cleared. As a side effect of this, before every put the server
> needs to contact zk and get meta table location and read meta to get region
> locations of the table.
> {noformat}
> 2018-09-13 22:21:15,467 TRACE [htable-pool11-t1] client.MetaCache(283):
> Removed all cached region locations that map to
> root1-thinkpad-t440p,35811,1536857446588
> 2018-09-13 22:21:15,467 DEBUG [HTableFlushWorker-5]
> client.HTableMultiplexer$FlushWorker(632): Processed 1 put requests for
> root1-ThinkPad-T440p:35811 and 0 failed, latency for this send: 5
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.reader=1,bindAddress=root1-ThinkPad-T440p,port=35811]
> ipc.RpcServer$Connection(1954): RequestHeader call_id: 218 method_name: "Get"
> request_param: true priority: 0 timeout: 6 totalRequestSize: 137 bytes
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> ipc.CallRunner(105): callId: 218 service: ClientService methodName: Get size:
> 137 connection: 127.0.0.1:42338 executing as root1
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> ipc.RpcServer(2356): callId: 218 service: ClientService methodName: Get size:
> 137 connection: 127.0.0.1:42338 param: region=
> testHTableMultiplexer_1,,1536857451720.304d914b641a738624937c7f9b4d684f.,
> row=\x00\x00\x00\xC4 connection: 127.0.0.1:42338, response result {
> associated_cell_count: 1 stale: false } queueTime: 0 processingTime: 0
> totalTime: 0
> 2018-09-13 22:21:15,516 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> io.BoundedByteBufferPool(106): runningAverage=16384, totalCapacity=0,
> count=0, allocations=1
> 2018-09-13 22:21:15,516 TRACE [main] ipc.AbstractRpcClient(236): Call: Get,
> callTime: 2ms
> 2018-09-13 22:21:15,516 TRACE [main] client.ClientScanner(122): Scan
> table=hbase:meta,
> startRow=testHTableMultiplexer_1,\x00\x00\x00\xC5,99
> 2018-09-13 22:21:15,516 TRACE [main] client.ClientSmallReversedScanner(179):
> Advancing internal small scanner to startKey at
> 'testHTableMultiplexer_1,\x00\x00\x00\xC5,99'
> 2018-09-13 22:21:15,517 TRACE [main] client.ZooKeeperRegistry(59): Looking up
> meta region location in ZK,
> connection=org.apache.hadoop.hbase.client.ZooKeeperRegistry@599f571f
> {noformat}
> From the minicluster logs [^HTableMultiplexer1000Puts.UT.txt] one can see
> that the string "Removed all cached region locations that map" and "Looking
> up meta region location in ZK" are present for every put.
> *Analysis:*
> The problem occurs as we call the {{cleanServerCache}} method always clears
> the server cache in case tablename is null and exception is null. See
> [AsyncRequestFutureImpl.java#L918|https://github.com/apache/hbase/blob/5d14c1af65c02f4e87059337c35e4431505de91c/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFutureImpl.java#L918]
> {code:java}
> private void cleanServerCache(ServerName server, Throwable regionException) {
> if (tableName == null &&
> ClientExceptionsUtil.isMetaClearingException(regionException)) {
> // For multi-actions, we don't have a table name, but we want to make
> sure to clear the
> // cache in case there were location-related exceptions. We don't to
> clear the cache
> // for every possible exception that comes through, however.
>
[jira] [Commented] (FLINK-10388) RestClientTest sometimes fails with AssertionError
[
https://issues.apache.org/jira/browse/FLINK-10388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632560#comment-16632560
]
Ted Yu commented on FLINK-10388:
I don't see 'Network unreachable' error in FLINK-4052
> RestClientTest sometimes fails with AssertionError
> --
>
> Key: FLINK-10388
> URL: https://issues.apache.org/jira/browse/FLINK-10388
> Project: Flink
> Issue Type: Test
>Reporter: Ted Yu
>Priority: Minor
>
> Running the test on Linux I got:
> {code}
> testConnectionTimeout(org.apache.flink.runtime.rest.RestClientTest) Time
> elapsed: 1.918 sec <<< FAILURE!
> java.lang.AssertionError:
> Expected: an instance of
> org.apache.flink.shaded.netty4.io.netty.channel.ConnectTimeoutException
> but:
> Network is unreachable: /10.255.255.1:80> is a
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedSocketException
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> at org.junit.Assert.assertThat(Assert.java:956)
> at org.junit.Assert.assertThat(Assert.java:923)
> at
> org.apache.flink.runtime.rest.RestClientTest.testConnectionTimeout(RestClientTest.java:69)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-9048) LocalFlinkMiniClusterITCase#testLocalFlinkMiniClusterWithMultipleTaskManagers sometimes fails
[
https://issues.apache.org/jira/browse/FLINK-9048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated FLINK-9048:
--
Description:
As of commit e0bc37bef69f5376d03214578e9b95816add661b, I got the following :
{code}
testLocalFlinkMiniClusterWithMultipleTaskManagers(org.apache.flink.test.runtime.minicluster.LocalFlinkMiniClusterITCase)
Time elapsed: 41.681 sec <<< FAILURE!
java.lang.AssertionError: Thread
Thread[ForkJoinPool.commonPool-worker-25,5,main] was started by the mini
cluster, but not shut down
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.flink.test.runtime.minicluster.LocalFlinkMiniClusterITCase.testLocalFlinkMiniClusterWithMultipleTaskManagers(LocalFlinkMiniClusterITCase.java:174)
{code}
was:
As of commit e0bc37bef69f5376d03214578e9b95816add661b, I got the following :
{code}
testLocalFlinkMiniClusterWithMultipleTaskManagers(org.apache.flink.test.runtime.minicluster.LocalFlinkMiniClusterITCase)
Time elapsed: 41.681 sec <<< FAILURE!
java.lang.AssertionError: Thread
Thread[ForkJoinPool.commonPool-worker-25,5,main] was started by the mini
cluster, but not shut down
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.flink.test.runtime.minicluster.LocalFlinkMiniClusterITCase.testLocalFlinkMiniClusterWithMultipleTaskManagers(LocalFlinkMiniClusterITCase.java:174)
{code}
> LocalFlinkMiniClusterITCase#testLocalFlinkMiniClusterWithMultipleTaskManagers
> sometimes fails
> -
>
> Key: FLINK-9048
> URL: https://issues.apache.org/jira/browse/FLINK-9048
> Project: Flink
> Issue Type: Test
>Reporter: Ted Yu
>Priority: Minor
>
> As of commit e0bc37bef69f5376d03214578e9b95816add661b, I got the following :
> {code}
> testLocalFlinkMiniClusterWithMultipleTaskManagers(org.apache.flink.test.runtime.minicluster.LocalFlinkMiniClusterITCase)
> Time elapsed: 41.681 sec <<< FAILURE!
> java.lang.AssertionError: Thread
> Thread[ForkJoinPool.commonPool-worker-25,5,main] was started by the mini
> cluster, but not shut down
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.test.runtime.minicluster.LocalFlinkMiniClusterITCase.testLocalFlinkMiniClusterWithMultipleTaskManagers(LocalFlinkMiniClusterITCase.java:174)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21216) TestSnapshotFromMaster#testSnapshotHFileArchiving is flaky
[
https://issues.apache.org/jira/browse/HBASE-21216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632510#comment-16632510
]
Ted Yu commented on HBASE-21216:
Looped the test 30 times locally with patch which all passed.
> TestSnapshotFromMaster#testSnapshotHFileArchiving is flaky
> --
>
> Key: HBASE-21216
> URL: https://issues.apache.org/jira/browse/HBASE-21216
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 21216.v1.txt
>
>
> From
> https://builds.apache.org/job/HBase-Flaky-Tests/job/branch-2/794/testReport/junit/org.apache.hadoop.hbase.master.cleaner/TestSnapshotFromMaster/testSnapshotHFileArchiving/
> :
> {code}
> java.lang.AssertionError: Archived hfiles [] and table hfiles
> [9ca09392705f425f9c916beedc10d63c] is missing snapshot
> file:6739a09747e54189a4112a6d8f37e894
> at
> org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster.testSnapshotHFileArchiving(TestSnapshotFromMaster.java:370)
> {code}
> The file appeared in archive dir before hfile cleaners were run:
> {code}
> 2018-09-20 10:38:53,187 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |-archive/
> 2018-09-20 10:38:53,188 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |data/
> 2018-09-20 10:38:53,189 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |---default/
> 2018-09-20 10:38:53,190 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |--test/
> 2018-09-20 10:38:53,191 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |-1237d57b63a7bdf067a930441a02514a/
> 2018-09-20 10:38:53,192 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |recovered.edits/
> 2018-09-20 10:38:53,193 DEBUG [Time-limited test] util.CommonFSUtils(774):
> |---4.seqid
> 2018-09-20 10:38:53,193 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |-29e1700e09b51223ad2f5811105a4d51/
> 2018-09-20 10:38:53,194 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |fam/
> 2018-09-20 10:38:53,195 DEBUG [Time-limited test] util.CommonFSUtils(774):
> |---2c66a18f6c1a4074b84ffbb3245268c4
> 2018-09-20 10:38:53,196 DEBUG [Time-limited test] util.CommonFSUtils(774):
> |---45bb396c6a5e49629e45a4d56f1e9b14
> 2018-09-20 10:38:53,196 DEBUG [Time-limited test] util.CommonFSUtils(774):
> |---6739a09747e54189a4112a6d8f37e894
> {code}
> However, the archive dir became empty after hfile cleaners were run:
> {code}
> 2018-09-20 10:38:53,312 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |-archive/
> 2018-09-20 10:38:53,313 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |-corrupt/
> {code}
> Leading to the assertion failure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-21216) TestSnapshotFromMaster#testSnapshotHFileArchiving is flaky
[
https://issues.apache.org/jira/browse/HBASE-21216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu reassigned HBASE-21216:
--
Assignee: Ted Yu
> TestSnapshotFromMaster#testSnapshotHFileArchiving is flaky
> --
>
> Key: HBASE-21216
> URL: https://issues.apache.org/jira/browse/HBASE-21216
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 21216.v1.txt
>
>
> From
> https://builds.apache.org/job/HBase-Flaky-Tests/job/branch-2/794/testReport/junit/org.apache.hadoop.hbase.master.cleaner/TestSnapshotFromMaster/testSnapshotHFileArchiving/
> :
> {code}
> java.lang.AssertionError: Archived hfiles [] and table hfiles
> [9ca09392705f425f9c916beedc10d63c] is missing snapshot
> file:6739a09747e54189a4112a6d8f37e894
> at
> org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster.testSnapshotHFileArchiving(TestSnapshotFromMaster.java:370)
> {code}
> The file appeared in archive dir before hfile cleaners were run:
> {code}
> 2018-09-20 10:38:53,187 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |-archive/
> 2018-09-20 10:38:53,188 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |data/
> 2018-09-20 10:38:53,189 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |---default/
> 2018-09-20 10:38:53,190 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |--test/
> 2018-09-20 10:38:53,191 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |-1237d57b63a7bdf067a930441a02514a/
> 2018-09-20 10:38:53,192 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |recovered.edits/
> 2018-09-20 10:38:53,193 DEBUG [Time-limited test] util.CommonFSUtils(774):
> |---4.seqid
> 2018-09-20 10:38:53,193 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |-29e1700e09b51223ad2f5811105a4d51/
> 2018-09-20 10:38:53,194 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |fam/
> 2018-09-20 10:38:53,195 DEBUG [Time-limited test] util.CommonFSUtils(774):
> |---2c66a18f6c1a4074b84ffbb3245268c4
> 2018-09-20 10:38:53,196 DEBUG [Time-limited test] util.CommonFSUtils(774):
> |---45bb396c6a5e49629e45a4d56f1e9b14
> 2018-09-20 10:38:53,196 DEBUG [Time-limited test] util.CommonFSUtils(774):
> |---6739a09747e54189a4112a6d8f37e894
> {code}
> However, the archive dir became empty after hfile cleaners were run:
> {code}
> 2018-09-20 10:38:53,312 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |-archive/
> 2018-09-20 10:38:53,313 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |-corrupt/
> {code}
> Leading to the assertion failure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21247) Allow WAL Provider to be specified by configuration without explicit enum in Providers
[ https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21247: --- Attachment: 21247.v4.txt > Allow WAL Provider to be specified by configuration without explicit enum in > Providers > -- > > Key: HBASE-21247 > URL: https://issues.apache.org/jira/browse/HBASE-21247 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: 3.0.0 > > Attachments: 21247.v1.txt, 21247.v2.txt, 21247.v3.txt, 21247.v4.txt > > > Currently all the WAL Providers acceptable to hbase are specified in > Providers enum of WALFactory. > This restricts the ability for additional WAL Providers to be supplied - by > class name. > This issue introduces additional config which allows the specification of new > WAL Provider through class name. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21186) Document hbase.regionserver.executor.openregion.threads in MTTR section
[ https://issues.apache.org/jira/browse/HBASE-21186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631326#comment-16631326 ] Ted Yu commented on HBASE-21186: bq. where single region is holding I think you meant single region server. Please check the grammar of your addition. Thanks > Document hbase.regionserver.executor.openregion.threads in MTTR section > --- > > Key: HBASE-21186 > URL: https://issues.apache.org/jira/browse/HBASE-21186 > Project: HBase > Issue Type: Improvement > Components: documentation >Reporter: Sahil Aggarwal >Assignee: Sahil Aggarwal >Priority: Minor > Attachments: HBASE-21186.master.001.patch > > > hbase.regionserver.executor.openregion.threads helps in improving MTTR by > increasing assign rpc processing rate at RS from HMaster but is not > documented. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMBARI-18952) Register BackupObserver and BackupHFileCleaner
[ https://issues.apache.org/jira/browse/AMBARI-18952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated AMBARI-18952: Description: Over in HBASE-14417, two new classes are added. org.apache.hadoop.hbase.backup.BackupHFileCleaner should be registered through hbase.master.hfilecleaner.plugins . It is responsible for keeping bulk loaded hfiles so that incremental backup can pick them up. org.apache.hadoop.hbase.backup.BackupObserver should be registered through hbase.coprocessor.region.classes It is notified when bulk load completes and writes records into hbase:backup table. was: Over in HBASE-14417, two new classes are added. org.apache.hadoop.hbase.backup.BackupHFileCleaner should be registered through hbase.master.hfilecleaner.plugins . It is responsible for keeping bulk loaded hfiles so that incremental backup can pick them up. org.apache.hadoop.hbase.backup.BackupObserver should be registered through hbase.coprocessor.region.classes It is notified when bulk load completes and writes records into hbase:backup table. > Register BackupObserver and BackupHFileCleaner > -- > > Key: AMBARI-18952 > URL: https://issues.apache.org/jira/browse/AMBARI-18952 > Project: Ambari > Issue Type: Improvement >Reporter: Ted Yu >Priority: Major > > Over in HBASE-14417, two new classes are added. > org.apache.hadoop.hbase.backup.BackupHFileCleaner should be registered > through hbase.master.hfilecleaner.plugins . It is responsible for keeping > bulk loaded hfiles so that incremental backup can pick them up. > org.apache.hadoop.hbase.backup.BackupObserver should be registered through > hbase.coprocessor.region.classes > It is notified when bulk load completes and writes records into hbase:backup > table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMBARI-13671) Ambari should check for duplicate config values
[
https://issues.apache.org/jira/browse/AMBARI-13671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated AMBARI-13671:
Description:
Using /#/main/services/HBASE/configs, I was able to save duplicate values for
hbase.coprocessor.region.classes :
{code}
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint,org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
{code}
Ambari should check for duplicate config values.
was:
Using /#/main/services/HBASE/configs, I was able to save duplicate values for
hbase.coprocessor.region.classes :
{code}
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint,org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
{code}
Ambari should check for duplicate config values.
> Ambari should check for duplicate config values
> ---
>
> Key: AMBARI-13671
> URL: https://issues.apache.org/jira/browse/AMBARI-13671
> Project: Ambari
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Major
>
> Using /#/main/services/HBASE/configs, I was able to save duplicate values for
> hbase.coprocessor.region.classes :
> {code}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint,org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> {code}
> Ambari should check for duplicate config values.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AMBARI-14160) Ambari Slider View should pick up multiple versions of the same app package
[ https://issues.apache.org/jira/browse/AMBARI-14160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated AMBARI-14160: Description: I was validating some fix for Slider hbase with Gour. A new Slider hbase app package was built and moved to under: /var/lib/ambari-server/resources/apps We tried naming the new app package ending with -ted.zip -999.zip After 'ambari-server restart', the new app package was not picked up by Ambari. Ambari Slider View should be able to accommodate more than one version of app package. was: I was validating some fix for Slider hbase with Gour. A new Slider hbase app package was built and moved to under: /var/lib/ambari-server/resources/apps We tried naming the new app package ending with -ted.zip -999.zip After 'ambari-server restart', the new app package was not picked up by Ambari. Ambari Slider View should be able to accommodate more than one version of app package. > Ambari Slider View should pick up multiple versions of the same app package > --- > > Key: AMBARI-14160 > URL: https://issues.apache.org/jira/browse/AMBARI-14160 > Project: Ambari > Issue Type: Improvement >Reporter: Ted Yu >Priority: Major > > I was validating some fix for Slider hbase with Gour. > A new Slider hbase app package was built and moved to under: > /var/lib/ambari-server/resources/apps > We tried naming the new app package ending with -ted.zip -999.zip > After 'ambari-server restart', the new app package was not picked up by > Ambari. > Ambari Slider View should be able to accommodate more than one version of app > package. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21247) Allow WAL Provider to be specified by configuration without explicit enum in Providers
[
https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631216#comment-16631216
]
Ted Yu commented on HBASE-21247:
>From https://builds.apache.org/job/PreCommit-HBASE-Build/14515/console :
{code}
22:32:24 [Thu Sep 27 22:32:24 UTC 2018 DEBUG]: jira_http_fetch:
https://issues.apache.org/jira/browse/HBASE-21247 returned 4xx status code.
Maybe incorrect username/password?
22:32:24 [Thu Sep 27 22:32:24 UTC 2018 DEBUG]: jira_locate_patch: not a JIRA.
{code}
> Allow WAL Provider to be specified by configuration without explicit enum in
> Providers
> --
>
> Key: HBASE-21247
> URL: https://issues.apache.org/jira/browse/HBASE-21247
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 21247.v1.txt, 21247.v2.txt, 21247.v3.txt
>
>
> Currently all the WAL Providers acceptable to hbase are specified in
> Providers enum of WALFactory.
> This restricts the ability for additional WAL Providers to be supplied - by
> class name.
> This issue introduces additional config which allows the specification of new
> WAL Provider through class name.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21247) Allow WAL Provider to be specified by configuration without explicit enum in Providers
[
https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631058#comment-16631058
]
Ted Yu commented on HBASE-21247:
>From https://builds.apache.org/job/PreCommit-HBASE-Build/14513/console :
{code}
20:10:21 ERROR: Unsure how to process HBASE-21247.
{code}
> Allow WAL Provider to be specified by configuration without explicit enum in
> Providers
> --
>
> Key: HBASE-21247
> URL: https://issues.apache.org/jira/browse/HBASE-21247
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 21247.v1.txt, 21247.v2.txt, 21247.v3.txt
>
>
> Currently all the WAL Providers acceptable to hbase are specified in
> Providers enum of WALFactory.
> This restricts the ability for additional WAL Providers to be supplied - by
> class name.
> This issue introduces additional config which allows the specification of new
> WAL Provider through class name.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21247) Allow WAL Provider to be specified by configuration without explicit enum in Providers
[ https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631035#comment-16631035 ] Ted Yu commented on HBASE-21247: In patch v3, there are two new subtests. * when only WALFactory.WAL_PROVIDER_CLASS is specified, verify that both wal provider and meta wal provider are of this class * when only WALFactory.META_WAL_PROVIDER_CLASS is specified, verify that wal provider is default and that meta wal provider is of this class > Allow WAL Provider to be specified by configuration without explicit enum in > Providers > -- > > Key: HBASE-21247 > URL: https://issues.apache.org/jira/browse/HBASE-21247 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: 3.0.0 > > Attachments: 21247.v1.txt, 21247.v2.txt, 21247.v3.txt > > > Currently all the WAL Providers acceptable to hbase are specified in > Providers enum of WALFactory. > This restricts the ability for additional WAL Providers to be supplied - by > class name. > This issue introduces additional config which allows the specification of new > WAL Provider through class name. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21247) Allow WAL Provider to be specified by configuration without explicit enum in Providers
[ https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21247: --- Attachment: 21247.v3.txt > Allow WAL Provider to be specified by configuration without explicit enum in > Providers > -- > > Key: HBASE-21247 > URL: https://issues.apache.org/jira/browse/HBASE-21247 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: 3.0.0 > > Attachments: 21247.v1.txt, 21247.v2.txt, 21247.v3.txt > > > Currently all the WAL Providers acceptable to hbase are specified in > Providers enum of WALFactory. > This restricts the ability for additional WAL Providers to be supplied - by > class name. > This issue introduces additional config which allows the specification of new > WAL Provider through class name. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21247) Allow WAL Provider to be specified by configuration without explicit enum in Providers
[ https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21247: --- Attachment: 21247.v2.txt > Allow WAL Provider to be specified by configuration without explicit enum in > Providers > -- > > Key: HBASE-21247 > URL: https://issues.apache.org/jira/browse/HBASE-21247 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: 3.0.0 > > Attachments: 21247.v1.txt, 21247.v2.txt > > > Currently all the WAL Providers acceptable to hbase are specified in > Providers enum of WALFactory. > This restricts the ability for additional WAL Providers to be supplied - by > class name. > This issue introduces additional config which allows the specification of new > WAL Provider through class name. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21247) Allow WAL Provider to be specified by configuration without explicit enum in Providers
[ https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21247: --- Release Note: Two config parameters, hbase.wal.provider.class and hbase.wal.meta_provider.class are introduced. hbase.wal.provider.class, when specified, configures the WAL provider class through its class name. If not specified, we fall back to the WAL provider enum specification. hbase.wal.meta_provider.class, when specified, configures the WAL provider class for hbase:meta through its class name. If not specified, we fall back to using the value for hbase.wal.provider.class . These new configs, when specified, override the enum WAL provider config. was: Two config parameters, hbase.wal.provider.class and hbase.wal.meta_provider.class are introduced. hbase.wal.provider.class, when specified, configures the WAL provider class through its class name. If not specified, we fall back to the WAL provider enum specification. hbase.wal.meta_provider.class, when specified, configures the WAL provider class for hbase:meta through its class name. If not specified, we fall back to using the value for hbase.wal.provider.class . > Allow WAL Provider to be specified by configuration without explicit enum in > Providers > -- > > Key: HBASE-21247 > URL: https://issues.apache.org/jira/browse/HBASE-21247 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: 3.0.0 > > Attachments: 21247.v1.txt > > > Currently all the WAL Providers acceptable to hbase are specified in > Providers enum of WALFactory. > This restricts the ability for additional WAL Providers to be supplied - by > class name. > This issue introduces additional config which allows the specification of new > WAL Provider through class name. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21247) Allow WAL Provider to be specified by configuration without explicit enum in Providers
[ https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21247: --- Release Note: Two config parameters, hbase.wal.provider.class and hbase.wal.meta_provider.class are introduced. hbase.wal.provider.class, when specified, configures the WAL provider class through its class name. If not specified, we fall back to the WAL provider enum specification. hbase.wal.meta_provider.class, when specified, configures the WAL provider class for hbase:meta through its class name. If not specified, we fall back to using the value for hbase.wal.provider.class . > Allow WAL Provider to be specified by configuration without explicit enum in > Providers > -- > > Key: HBASE-21247 > URL: https://issues.apache.org/jira/browse/HBASE-21247 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: 3.0.0 > > Attachments: 21247.v1.txt > > > Currently all the WAL Providers acceptable to hbase are specified in > Providers enum of WALFactory. > This restricts the ability for additional WAL Providers to be supplied - by > class name. > This issue introduces additional config which allows the specification of new > WAL Provider through class name. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21247) Allow WAL Provider to be specified by configuration without explicit enum in Providers
[
https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630925#comment-16630925
]
Ted Yu commented on HBASE-21247:
w.r.t. first comment above,
{code}
boolean metaWALProvPresent = conf.get(META_WAL_PROVIDER_CLASS) != null;
provider = createProvider(getProviderClass(
metaWALProvPresent ? META_WAL_PROVIDER_CLASS : WAL_PROVIDER_CLASS,
{code}
when META_WAL_PROVIDER_CLASS is not specified, we fall back to the value for
WAL_PROVIDER_CLASS (if present).
The user doesn't need to specify two WAL classes in the normal case.
> Allow WAL Provider to be specified by configuration without explicit enum in
> Providers
> --
>
> Key: HBASE-21247
> URL: https://issues.apache.org/jira/browse/HBASE-21247
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: 21247.v1.txt
>
>
> Currently all the WAL Providers acceptable to hbase are specified in
> Providers enum of WALFactory.
> This restricts the ability for additional WAL Providers to be supplied - by
> class name.
> This issue introduces additional config which allows the specification of new
> WAL Provider through class name.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21200) Memstore flush doesn't finish because of seekToPreviousRow() in memstore scanner.
[ https://issues.apache.org/jira/browse/HBASE-21200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630858#comment-16630858 ] Ted Yu commented on HBASE-21200: I ran the new test on master branch which passed. What error is expected from the new test ? > Memstore flush doesn't finish because of seekToPreviousRow() in memstore > scanner. > - > > Key: HBASE-21200 > URL: https://issues.apache.org/jira/browse/HBASE-21200 > Project: HBase > Issue Type: Bug > Components: Scanners >Reporter: dongjin2193.jeon >Priority: Major > Attachments: HBASE-21200-UT.patch, RegionServerJstack.log > > > The issue of delaying memstore flush still occurs after backport hbase-15871. > Reverse scan takes a long time to seek previous row in the memstore full of > deleted cells. > > jstack : > "MemStoreFlusher.0" #114 prio=5 os_prio=0 tid=0x7fa3d0729000 nid=0x486a > waiting on condition [0x7fa3b9b6b000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0xa465fe60> (a > java.util.concurrent.locks.ReentrantLock$NonfairSync) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) > at > java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) > at > java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) > at > org.apache.hadoop.hbase.regionserver.*StoreScanner.updateReaders(StoreScanner.java:695)* > at > org.apache.hadoop.hbase.regionserver.HStore.notifyChangedReadersObservers(HStore.java:1127) > at > org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1106) > at > org.apache.hadoop.hbase.regionserver.HStore.access$600(HStore.java:130) > at > org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2455) > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2519) > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2256) > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2218) > at > org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2110) > at > org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:2036) > at > org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:501) > at > org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:471) > at > org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$800(MemStoreFlusher.java:75) > at > org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:259) > at java.lang.Thread.run(Thread.java:748) > > "RpcServer.FifoWFPBQ.default.handler=27,queue=0,port=16020" #65 daemon prio=5 > os_prio=0 tid=0x7fa3e628 nid=0x4801 runnable [0x7fa3bd29a000] > java.lang.Thread.State: RUNNABLE > at > org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner.getNext(DefaultMemStore.java:780) > at > org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner.seekInSubLists(DefaultMemStore.java:826) > - locked <0xb45aa5b8> (a > org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner) > at > org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner.seek(DefaultMemStore.java:818) > - locked <0xb45aa5b8> (a > org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner) > at > org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner.seekToPreviousRow(DefaultMemStore.java:1000) > - locked <0xb45aa5b8> (a > org.apache.hadoop.hbase.regionserver.DefaultMemStore$MemStoreScanner) > at > org.apache.hadoop.hbase.regionserver.ReversedKeyValueHeap.next(ReversedKeyValueHeap.java:136) > at > org.apache.hadoop.hbase.regionserver.*StoreScanner.next(StoreScanner.java:629)* > at > org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:147) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5876) > at >
[jira] [Commented] (KYLIN-3584) Incorrect lazy initialization of DefaultScheduler.INSTANCE
[
https://issues.apache.org/jira/browse/KYLIN-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630845#comment-16630845
]
Ted Yu commented on KYLIN-3584:
---
DefaultScheduler#init is not involved in {{createInstance}}.
What happens when an un-initialized INSTANCE is handed to the caller of
getInstance ?
> Incorrect lazy initialization of DefaultScheduler.INSTANCE
> --
>
> Key: KYLIN-3584
> URL: https://issues.apache.org/jira/browse/KYLIN-3584
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>
> Here is related code:
> {code}
> public static DefaultScheduler getInstance() {
> if (INSTANCE == null) {
> INSTANCE = createInstance();
> {code}
> The check against may not be effective.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21247) Allow WAL Provider to be specified by configuration without explicit enum in Providers
[ https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21247: --- Status: Patch Available (was: Open) > Allow WAL Provider to be specified by configuration without explicit enum in > Providers > -- > > Key: HBASE-21247 > URL: https://issues.apache.org/jira/browse/HBASE-21247 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 21247.v1.txt > > > Currently all the WAL Providers acceptable to hbase are specified in > Providers enum of WALFactory. > This restricts the ability for additional WAL Providers to be supplied - by > class name. > This issue introduces additional config which allows the specification of new > WAL Provider through class name. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21247) Allow WAL Provider to be specified by configuration without explicit enum in Providers
Ted Yu created HBASE-21247: -- Summary: Allow WAL Provider to be specified by configuration without explicit enum in Providers Key: HBASE-21247 URL: https://issues.apache.org/jira/browse/HBASE-21247 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: Ted Yu Attachments: 21247.v1.txt Currently all the WAL Providers acceptable to hbase are specified in Providers enum of WALFactory. This restricts the ability for additional WAL Providers to be supplied - by class name. This issue introduces additional config which allows the specification of new WAL Provider through class name. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21247) Allow WAL Provider to be specified by configuration without explicit enum in Providers
Ted Yu created HBASE-21247: -- Summary: Allow WAL Provider to be specified by configuration without explicit enum in Providers Key: HBASE-21247 URL: https://issues.apache.org/jira/browse/HBASE-21247 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: Ted Yu Attachments: 21247.v1.txt Currently all the WAL Providers acceptable to hbase are specified in Providers enum of WALFactory. This restricts the ability for additional WAL Providers to be supplied - by class name. This issue introduces additional config which allows the specification of new WAL Provider through class name. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21247) Allow WAL Provider to be specified by configuration without explicit enum in Providers
[ https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21247: --- Attachment: 21247.v1.txt > Allow WAL Provider to be specified by configuration without explicit enum in > Providers > -- > > Key: HBASE-21247 > URL: https://issues.apache.org/jira/browse/HBASE-21247 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 21247.v1.txt > > > Currently all the WAL Providers acceptable to hbase are specified in > Providers enum of WALFactory. > This restricts the ability for additional WAL Providers to be supplied - by > class name. > This issue introduces additional config which allows the specification of new > WAL Provider through class name. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21246) Introduce WALIdentity interface
[ https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21246: --- Status: Patch Available (was: Open) > Introduce WALIdentity interface > --- > > Key: HBASE-21246 > URL: https://issues.apache.org/jira/browse/HBASE-21246 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 21246.HBASE-20952.001.patch > > > We are introducing WALIdentity interface so that the WAL representation can > be decoupled from distributed filesystem. > The interface provides getName method whose return value can represent > filename in distributed filesystem environment or, the name of the stream > when the WAL is backed by log stream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21246) Introduce WALIdentity interface
[ https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21246: --- Attachment: 21246.HBASE-20952.001.patch > Introduce WALIdentity interface > --- > > Key: HBASE-21246 > URL: https://issues.apache.org/jira/browse/HBASE-21246 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 21246.HBASE-20952.001.patch > > > We are introducing WALIdentity interface so that the WAL representation can > be decoupled from distributed filesystem. > The interface provides getName method whose return value can represent > filename in distributed filesystem environment or, the name of the stream > when the WAL is backed by log stream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21246) Introduce WALIdentity interface
Ted Yu created HBASE-21246: -- Summary: Introduce WALIdentity interface Key: HBASE-21246 URL: https://issues.apache.org/jira/browse/HBASE-21246 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Assignee: Ted Yu We are introducing WALIdentity interface so that the WAL representation can be decoupled from distributed filesystem. The interface provides getName method whose return value can represent filename in distributed filesystem environment or, the name of the stream when the WAL is backed by log stream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21246) Introduce WALIdentity interface
Ted Yu created HBASE-21246: -- Summary: Introduce WALIdentity interface Key: HBASE-21246 URL: https://issues.apache.org/jira/browse/HBASE-21246 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Assignee: Ted Yu We are introducing WALIdentity interface so that the WAL representation can be decoupled from distributed filesystem. The interface provides getName method whose return value can represent filename in distributed filesystem environment or, the name of the stream when the WAL is backed by log stream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20952) Re-visit the WAL API
[
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20952:
---
Issue Type: Improvement (was: Sub-task)
Parent: (was: HBASE-20951)
> Re-visit the WAL API
>
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Josh Elser
>Priority: Major
> Attachments: 20952.v1.txt
>
>
> Take a step back from the current WAL implementations and think about what an
> HBase WAL API should look like. What are the primitive calls that we require
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We
> should also have a mind for what is happening in the Ratis LogService (but
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and
> backup Replication has the use-case for "tail"'ing the WAL which we
> should provide via our new API. B doesn't do anything fancy (IIRC). We
> should make sure all consumers are generally going to be OK with the API we
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods
> which were "bolted" on such as {{AbstractFSWAL}} and
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability
> annotations are chosen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3586) Boxing/unboxing to parse a primitive is suboptimal
[
https://issues.apache.org/jira/browse/KYLIN-3586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630670#comment-16630670
]
Ted Yu commented on KYLIN-3586:
---
There are still several {{Integer.valueOf}} calls in the code base.
Can you address them as well ?
Also, the commit log didn't contain JIRA number.
> Boxing/unboxing to parse a primitive is suboptimal
> --
>
> Key: KYLIN-3586
> URL: https://issues.apache.org/jira/browse/KYLIN-3586
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Lijun Cao
>Priority: Major
> Fix For: v2.6.0
>
>
> An example is from HBaseLookupRowEncoder:
> {code}
> int valIdx = Integer.valueOf(Bytes.toString(qualifier));
> {code}
> valueOf returns an Integer object which would be unboxed and assigned to
> valIdx.
> Integer.parseInt() should be used instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3598) Use the "guava beta checker" plugin to stay away from @Beta API
[ https://issues.apache.org/jira/browse/KYLIN-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated KYLIN-3598: -- Summary: Use the "guava beta checker" plugin to stay away from @Beta API (was: Use the "guava beta checker" plugin to keep off of @Beta stuff) > Use the "guava beta checker" plugin to stay away from @Beta API > --- > > Key: KYLIN-3598 > URL: https://issues.apache.org/jira/browse/KYLIN-3598 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Priority: Major > > The Guava people publish an Error Prone plugin to detect when stuff that's > annotated with @Beta gets used. Those things shouldn't be used because the > project gives no promises about deprecating before removal. > plugin: > https://github.com/google/guava-beta-checker -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-10446) Use the "guava beta checker" plugin to keep off of @Beta API
Ted Yu created FLINK-10446: -- Summary: Use the "guava beta checker" plugin to keep off of @Beta API Key: FLINK-10446 URL: https://issues.apache.org/jira/browse/FLINK-10446 Project: Flink Issue Type: Task Reporter: Ted Yu The Guava people publish an Error Prone plugin to detect when stuff that's annotated with @Beta gets used. Those things shouldn't be used because the project gives no promises about deprecating before removal. plugin: https://github.com/google/guava-beta-checker -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-10446) Use the "guava beta checker" plugin to keep off of @Beta API
Ted Yu created FLINK-10446: -- Summary: Use the "guava beta checker" plugin to keep off of @Beta API Key: FLINK-10446 URL: https://issues.apache.org/jira/browse/FLINK-10446 Project: Flink Issue Type: Task Reporter: Ted Yu The Guava people publish an Error Prone plugin to detect when stuff that's annotated with @Beta gets used. Those things shouldn't be used because the project gives no promises about deprecating before removal. plugin: https://github.com/google/guava-beta-checker -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ZOOKEEPER-1936) Server exits when unable to create data directory due to race
[ https://issues.apache.org/jira/browse/ZOOKEEPER-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629504#comment-16629504 ] Ted Yu commented on ZOOKEEPER-1936: --- Can you outline how you plan to fix ? thanks > Server exits when unable to create data directory due to race > -- > > Key: ZOOKEEPER-1936 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1936 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.6, 3.5.0 >Reporter: Harald Musum >Assignee: Ted Yu >Priority: Minor > Fix For: 3.6.0, 3.5.5 > > Attachments: ZOOKEEPER-1936.branch-3.4.patch, ZOOKEEPER-1936.patch, > ZOOKEEPER-1936.v2.patch, ZOOKEEPER-1936.v3.patch, ZOOKEEPER-1936.v3.patch, > ZOOKEEPER-1936.v4.patch, ZOOKEEPER-1936.v5.patch > > > We sometime see issues with ZooKeeper server not starting and seeing this > error in the log: > [2014-05-27 09:29:48.248] ERROR : - > .org.apache.zookeeper.server.ZooKeeperServerMainUnexpected exception, > exiting abnormally\nexception=\njava.io.IOException: Unable to create data > directory /home/y/var/zookeeper/version-2\n\tat > org.apache.zookeeper.server.persistence.FileTxnSnapLog.(FileTxnSnapLog.java:85)\n\tat > org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:103)\n\tat > org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86)\n\tat > org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52)\n\tat > org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)\n\tat > org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)\n\t > [...] > Stack trace from JVM gives this: > "PurgeTask" daemon prio=10 tid=0x0201d000 nid=0x1727 runnable > [0x7f55d7dc7000] >java.lang.Thread.State: RUNNABLE > at java.io.UnixFileSystem.createDirectory(Native Method) > at java.io.File.mkdir(File.java:1310) > at java.io.File.mkdirs(File.java:1337) > at > org.apache.zookeeper.server.persistence.FileTxnSnapLog.(FileTxnSnapLog.java:84) > at org.apache.zookeeper.server.PurgeTxnLog.purge(PurgeTxnLog.java:68) > at > org.apache.zookeeper.server.DatadirCleanupManager$PurgeTask.run(DatadirCleanupManager.java:140) > at java.util.TimerThread.mainLoop(Timer.java:555) > at java.util.TimerThread.run(Timer.java:505) > "zookeeper server" prio=10 tid=0x027df800 nid=0x1715 runnable > [0x7f55d7ed8000] >java.lang.Thread.State: RUNNABLE > at java.io.UnixFileSystem.createDirectory(Native Method) > at java.io.File.mkdir(File.java:1310) > at java.io.File.mkdirs(File.java:1337) > at > org.apache.zookeeper.server.persistence.FileTxnSnapLog.(FileTxnSnapLog.java:84) > at > org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:103) > at > org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86) > at > org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52) > at > org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116) > at > org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78) > [...] > So it seems that when autopurge is used (as it is in our case), it might > happen at the same time as starting the server itself. In FileTxnSnapLog() it > will check if the directory exists and create it if not. These two tasks do > this at the same time, and mkdir fails and server exits the JVM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (RATIS-195) TestRaftReconfigurationWithSimulatedRpc fails with assertion error
[
https://issues.apache.org/jira/browse/RATIS-195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved RATIS-195.
--
Resolution: Cannot Reproduce
> TestRaftReconfigurationWithSimulatedRpc fails with assertion error
> --
>
> Key: RATIS-195
> URL: https://issues.apache.org/jira/browse/RATIS-195
> Project: Ratis
> Issue Type: Test
> Components: test
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments:
> org.apache.ratis.server.simulation.TestRaftReconfigurationWithSimulatedRpc-output.txt,
> ratis-8464.tst, testRaftReconfigurationWithSimulatedRpc-linux-output.txt
>
>
> As of commit 7b3a9a6f5f8e8075727d84e3ddeae7b594eda89c, I observed the
> following :
> {code}
> testRevertConfigurationChange(org.apache.ratis.server.simulation.TestRaftReconfigurationWithSimulatedRpc)
> Time elapsed: 2.119 sec <<< FAILURE!
> java.lang.AssertionError: 1 0 expected: but was:
> {code}
> 1 was confIndex and 0 was log.getLastCommittedIndex()
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HDFS-6092) DistributedFileSystem#getCanonicalServiceName() and DistributedFileSystem#getUri() may return inconsistent results w.r.t. port
[
https://issues.apache.org/jira/browse/HDFS-6092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629503#comment-16629503
]
Ted Yu commented on HDFS-6092:
--
Test failure was not related.
> DistributedFileSystem#getCanonicalServiceName() and
> DistributedFileSystem#getUri() may return inconsistent results w.r.t. port
> --
>
> Key: HDFS-6092
> URL: https://issues.apache.org/jira/browse/HDFS-6092
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Labels: BB2015-05-TBR
> Attachments: HDFS-6092-v4.patch, HDFS-6092-v5.patch,
> haosdent-HDFS-6092-v2.patch, haosdent-HDFS-6092.patch, hdfs-6092-v1.txt,
> hdfs-6092-v2.txt, hdfs-6092-v3.txt
>
>
> I discovered this when working on HBASE-10717
> Here is sample code to reproduce the problem:
> {code}
> Path desPath = new Path("hdfs://127.0.0.1/");
> FileSystem desFs = desPath.getFileSystem(conf);
>
> String s = desFs.getCanonicalServiceName();
> URI uri = desFs.getUri();
> {code}
> Canonical name string contains the default port - 8020
> But uri doesn't contain port.
> This would result in the following exception:
> {code}
> testIsSameHdfs(org.apache.hadoop.hbase.util.TestFSHDFSUtils) Time elapsed:
> 0.001 sec <<< ERROR!
> java.lang.IllegalArgumentException: port out of range:-1
> at java.net.InetSocketAddress.checkPort(InetSocketAddress.java:143)
> at java.net.InetSocketAddress.(InetSocketAddress.java:224)
> at
> org.apache.hadoop.hbase.util.FSHDFSUtils.getNNAddresses(FSHDFSUtils.java:88)
> {code}
> Thanks to Brando Li who helped debug this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (KYLIN-3598) Use the "guava beta checker" plugin to keep off of @Beta stuff
Ted Yu created KYLIN-3598: - Summary: Use the "guava beta checker" plugin to keep off of @Beta stuff Key: KYLIN-3598 URL: https://issues.apache.org/jira/browse/KYLIN-3598 Project: Kylin Issue Type: Task Reporter: Ted Yu The Guava people publish an Error Prone plugin to detect when stuff that's annotated with @Beta gets used. Those things shouldn't be used because the project gives no promises about deprecating before removal. plugin: https://github.com/google/guava-beta-checker -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3598) Use the "guava beta checker" plugin to keep off of @Beta stuff
Ted Yu created KYLIN-3598: - Summary: Use the "guava beta checker" plugin to keep off of @Beta stuff Key: KYLIN-3598 URL: https://issues.apache.org/jira/browse/KYLIN-3598 Project: Kylin Issue Type: Task Reporter: Ted Yu The Guava people publish an Error Prone plugin to detect when stuff that's annotated with @Beta gets used. Those things shouldn't be used because the project gives no promises about deprecating before removal. plugin: https://github.com/google/guava-beta-checker -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-10228) Add metrics for netty direct memory consumption
[ https://issues.apache.org/jira/browse/FLINK-10228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated FLINK-10228: --- Description: netty direct memory usage can be exposed via metrics so that operator can keep track of memory consumption . (was: netty direct memory usage can be exposed via metrics so that operator can keep track of memory consumption.) > Add metrics for netty direct memory consumption > --- > > Key: FLINK-10228 > URL: https://issues.apache.org/jira/browse/FLINK-10228 > Project: Flink > Issue Type: Improvement > Components: Metrics >Reporter: Ted Yu >Assignee: vinoyang >Priority: Minor > > netty direct memory usage can be exposed via metrics so that operator can > keep track of memory consumption . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21216) TestSnapshotFromMaster#testSnapshotHFileArchiving is flaky
[
https://issues.apache.org/jira/browse/HBASE-21216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-21216:
---
Status: Patch Available (was: Open)
> TestSnapshotFromMaster#testSnapshotHFileArchiving is flaky
> --
>
> Key: HBASE-21216
> URL: https://issues.apache.org/jira/browse/HBASE-21216
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Priority: Major
> Attachments: 21216.v1.txt
>
>
> From
> https://builds.apache.org/job/HBase-Flaky-Tests/job/branch-2/794/testReport/junit/org.apache.hadoop.hbase.master.cleaner/TestSnapshotFromMaster/testSnapshotHFileArchiving/
> :
> {code}
> java.lang.AssertionError: Archived hfiles [] and table hfiles
> [9ca09392705f425f9c916beedc10d63c] is missing snapshot
> file:6739a09747e54189a4112a6d8f37e894
> at
> org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster.testSnapshotHFileArchiving(TestSnapshotFromMaster.java:370)
> {code}
> The file appeared in archive dir before hfile cleaners were run:
> {code}
> 2018-09-20 10:38:53,187 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |-archive/
> 2018-09-20 10:38:53,188 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |data/
> 2018-09-20 10:38:53,189 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |---default/
> 2018-09-20 10:38:53,190 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |--test/
> 2018-09-20 10:38:53,191 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |-1237d57b63a7bdf067a930441a02514a/
> 2018-09-20 10:38:53,192 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |recovered.edits/
> 2018-09-20 10:38:53,193 DEBUG [Time-limited test] util.CommonFSUtils(774):
> |---4.seqid
> 2018-09-20 10:38:53,193 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |-29e1700e09b51223ad2f5811105a4d51/
> 2018-09-20 10:38:53,194 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |fam/
> 2018-09-20 10:38:53,195 DEBUG [Time-limited test] util.CommonFSUtils(774):
> |---2c66a18f6c1a4074b84ffbb3245268c4
> 2018-09-20 10:38:53,196 DEBUG [Time-limited test] util.CommonFSUtils(774):
> |---45bb396c6a5e49629e45a4d56f1e9b14
> 2018-09-20 10:38:53,196 DEBUG [Time-limited test] util.CommonFSUtils(774):
> |---6739a09747e54189a4112a6d8f37e894
> {code}
> However, the archive dir became empty after hfile cleaners were run:
> {code}
> 2018-09-20 10:38:53,312 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |-archive/
> 2018-09-20 10:38:53,313 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |-corrupt/
> {code}
> Leading to the assertion failure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21216) TestSnapshotFromMaster#testSnapshotHFileArchiving is flaky
[
https://issues.apache.org/jira/browse/HBASE-21216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-21216:
---
Attachment: 21216.v1.txt
> TestSnapshotFromMaster#testSnapshotHFileArchiving is flaky
> --
>
> Key: HBASE-21216
> URL: https://issues.apache.org/jira/browse/HBASE-21216
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Priority: Major
> Attachments: 21216.v1.txt
>
>
> From
> https://builds.apache.org/job/HBase-Flaky-Tests/job/branch-2/794/testReport/junit/org.apache.hadoop.hbase.master.cleaner/TestSnapshotFromMaster/testSnapshotHFileArchiving/
> :
> {code}
> java.lang.AssertionError: Archived hfiles [] and table hfiles
> [9ca09392705f425f9c916beedc10d63c] is missing snapshot
> file:6739a09747e54189a4112a6d8f37e894
> at
> org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster.testSnapshotHFileArchiving(TestSnapshotFromMaster.java:370)
> {code}
> The file appeared in archive dir before hfile cleaners were run:
> {code}
> 2018-09-20 10:38:53,187 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |-archive/
> 2018-09-20 10:38:53,188 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |data/
> 2018-09-20 10:38:53,189 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |---default/
> 2018-09-20 10:38:53,190 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |--test/
> 2018-09-20 10:38:53,191 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |-1237d57b63a7bdf067a930441a02514a/
> 2018-09-20 10:38:53,192 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |recovered.edits/
> 2018-09-20 10:38:53,193 DEBUG [Time-limited test] util.CommonFSUtils(774):
> |---4.seqid
> 2018-09-20 10:38:53,193 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |-29e1700e09b51223ad2f5811105a4d51/
> 2018-09-20 10:38:53,194 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |fam/
> 2018-09-20 10:38:53,195 DEBUG [Time-limited test] util.CommonFSUtils(774):
> |---2c66a18f6c1a4074b84ffbb3245268c4
> 2018-09-20 10:38:53,196 DEBUG [Time-limited test] util.CommonFSUtils(774):
> |---45bb396c6a5e49629e45a4d56f1e9b14
> 2018-09-20 10:38:53,196 DEBUG [Time-limited test] util.CommonFSUtils(774):
> |---6739a09747e54189a4112a6d8f37e894
> {code}
> However, the archive dir became empty after hfile cleaners were run:
> {code}
> 2018-09-20 10:38:53,312 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |-archive/
> 2018-09-20 10:38:53,313 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |-corrupt/
> {code}
> Leading to the assertion failure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21216) TestSnapshotFromMaster#testSnapshotHFileArchiving is flaky
[
https://issues.apache.org/jira/browse/HBASE-21216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628994#comment-16628994
]
Ted Yu commented on HBASE-21216:
The cause for the assertion failure was that the trigger of refresh for
SnapshotFileCache is indeterministic w.r.t. the timing of the assertion.
If cache of SnapshotFileCache is refreshed right before the collection of
archive files, archives may be cleaned by the SnapshotHFileCleaner.
I have a patch using which I loop the test locally.
So far 12 iterations have passed successfully.
Will upload the patch in a moment.
> TestSnapshotFromMaster#testSnapshotHFileArchiving is flaky
> --
>
> Key: HBASE-21216
> URL: https://issues.apache.org/jira/browse/HBASE-21216
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Priority: Major
>
> From
> https://builds.apache.org/job/HBase-Flaky-Tests/job/branch-2/794/testReport/junit/org.apache.hadoop.hbase.master.cleaner/TestSnapshotFromMaster/testSnapshotHFileArchiving/
> :
> {code}
> java.lang.AssertionError: Archived hfiles [] and table hfiles
> [9ca09392705f425f9c916beedc10d63c] is missing snapshot
> file:6739a09747e54189a4112a6d8f37e894
> at
> org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster.testSnapshotHFileArchiving(TestSnapshotFromMaster.java:370)
> {code}
> The file appeared in archive dir before hfile cleaners were run:
> {code}
> 2018-09-20 10:38:53,187 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |-archive/
> 2018-09-20 10:38:53,188 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |data/
> 2018-09-20 10:38:53,189 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |---default/
> 2018-09-20 10:38:53,190 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |--test/
> 2018-09-20 10:38:53,191 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |-1237d57b63a7bdf067a930441a02514a/
> 2018-09-20 10:38:53,192 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |recovered.edits/
> 2018-09-20 10:38:53,193 DEBUG [Time-limited test] util.CommonFSUtils(774):
> |---4.seqid
> 2018-09-20 10:38:53,193 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |-29e1700e09b51223ad2f5811105a4d51/
> 2018-09-20 10:38:53,194 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |fam/
> 2018-09-20 10:38:53,195 DEBUG [Time-limited test] util.CommonFSUtils(774):
> |---2c66a18f6c1a4074b84ffbb3245268c4
> 2018-09-20 10:38:53,196 DEBUG [Time-limited test] util.CommonFSUtils(774):
> |---45bb396c6a5e49629e45a4d56f1e9b14
> 2018-09-20 10:38:53,196 DEBUG [Time-limited test] util.CommonFSUtils(774):
> |---6739a09747e54189a4112a6d8f37e894
> {code}
> However, the archive dir became empty after hfile cleaners were run:
> {code}
> 2018-09-20 10:38:53,312 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |-archive/
> 2018-09-20 10:38:53,313 DEBUG [Time-limited test] util.CommonFSUtils(771):
> |-corrupt/
> {code}
> Leading to the assertion failure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
