[jira] [Assigned] (HBASE-23864) No need to submit SplitTableRegionProcedure/MergeTableRegionsProcedure when split/merge is disabled
[ https://issues.apache.org/jira/browse/HBASE-23864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang reassigned HBASE-23864: -- Assignee: Guanghao Zhang > No need to submit SplitTableRegionProcedure/MergeTableRegionsProcedure when > split/merge is disabled > --- > > Key: HBASE-23864 > URL: https://issues.apache.org/jira/browse/HBASE-23864 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Attachments: HBASE-23864.PNG > > > Now even the split/merge is disabled, master will submit a > SplitTableRegionProcedure, too. And rollback it when execute failed. I > thought the split/merge switch is a cluster level swtich. Master can check it > early and no need to submit > SplitTableRegionProcedure/MergeTableRegionsProcedure when split/merge switch > is disabled. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23864) No need to submit SplitTableRegionProcedure/MergeTableRegionsProcedure when split/merge is disabled
[ https://issues.apache.org/jira/browse/HBASE-23864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-23864: --- Summary: No need to submit SplitTableRegionProcedure/MergeTableRegionsProcedure when split/merge is disabled (was: No need to submit SplitTableRegionProcedure/MergeTableRegionsProcedure when split/merge switch is disabled) > No need to submit SplitTableRegionProcedure/MergeTableRegionsProcedure when > split/merge is disabled > --- > > Key: HBASE-23864 > URL: https://issues.apache.org/jira/browse/HBASE-23864 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Priority: Major > Attachments: HBASE-23864.PNG > > > Now even the split/merge is disabled, master will submit a > SplitTableRegionProcedure, too. And rollback it when execute failed. I > thought the split/merge switch is a cluster level swtich. Master can check it > early and no need to submit > SplitTableRegionProcedure/MergeTableRegionsProcedure when split/merge switch > is disabled. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23864) No need to submit SplitTableRegionProcedure/MergeTableRegionsProcedure when split/merge switch is disabled
[ https://issues.apache.org/jira/browse/HBASE-23864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-23864: --- Attachment: HBASE-23864.PNG > No need to submit SplitTableRegionProcedure/MergeTableRegionsProcedure when > split/merge switch is disabled > -- > > Key: HBASE-23864 > URL: https://issues.apache.org/jira/browse/HBASE-23864 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Priority: Major > Attachments: HBASE-23864.PNG > > > Now even the split/merge is disabled, master will submit a > SplitTableRegionProcedure, too. And rollback it when execute failed. I > thought the split/merge switch is a cluster level swtich. Master can check it > early and no need to submit > SplitTableRegionProcedure/MergeTableRegionsProcedure when split/merge switch > is disabled. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23864) No need to submit SplitTableRegionProcedure/MergeTableRegionsProcedure when split/merge switch is disabled
Guanghao Zhang created HBASE-23864: -- Summary: No need to submit SplitTableRegionProcedure/MergeTableRegionsProcedure when split/merge switch is disabled Key: HBASE-23864 URL: https://issues.apache.org/jira/browse/HBASE-23864 Project: HBase Issue Type: Bug Reporter: Guanghao Zhang Now even the split/merge is disabled, master will submit a SplitTableRegionProcedure, too. And rollback it when execute failed. I thought the split/merge switch is a cluster level swtich. Master can check it early and no need to submit SplitTableRegionProcedure/MergeTableRegionsProcedure when split/merge switch is disabled. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23864) No need to submit SplitTableRegionProcedure/MergeTableRegionsProcedure when split/merge switch is disabled
[ https://issues.apache.org/jira/browse/HBASE-23864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-23864: --- Issue Type: Improvement (was: Bug) > No need to submit SplitTableRegionProcedure/MergeTableRegionsProcedure when > split/merge switch is disabled > -- > > Key: HBASE-23864 > URL: https://issues.apache.org/jira/browse/HBASE-23864 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Priority: Major > > Now even the split/merge is disabled, master will submit a > SplitTableRegionProcedure, too. And rollback it when execute failed. I > thought the split/merge switch is a cluster level swtich. Master can check it > early and no need to submit > SplitTableRegionProcedure/MergeTableRegionsProcedure when split/merge switch > is disabled. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23859) Modify "Block locality" of RegionServer Web UI to human readable percentage
[ https://issues.apache.org/jira/browse/HBASE-23859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang resolved HBASE-23859. Fix Version/s: 2.1.10 2.2.4 2.3.0 3.0.0 Resolution: Fixed Pushed to branch-2.1+. Thanks [~GeorryHuang] for contributing. > Modify "Block locality" of RegionServer Web UI to human readable percentage > --- > > Key: HBASE-23859 > URL: https://issues.apache.org/jira/browse/HBASE-23859 > Project: HBase > Issue Type: Improvement >Affects Versions: master >Reporter: Zhuoyue Huang >Assignee: Zhuoyue Huang >Priority: Trivial > Fix For: 3.0.0, 2.3.0, 2.2.4, 2.1.10 > > Attachments: image-2020-02-17-20-36-50-084.png, > image-2020-02-17-20-38-52-990.png > > > The unit of "Block locality" in Web UI just like picture 1 > !image-2020-02-17-20-36-50-084.png! > I think we should change it to percentage unit like picture 2 > !image-2020-02-17-20-38-52-990.png! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23855) Change bytes size to human readable size for Server Metrics of RegionServer Web UI
[ https://issues.apache.org/jira/browse/HBASE-23855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang resolved HBASE-23855. Resolution: Fixed Pushed to master and branch-2. Thanks [~GeorryHuang] for contributing. > Change bytes size to human readable size for Server Metrics of RegionServer > Web UI > -- > > Key: HBASE-23855 > URL: https://issues.apache.org/jira/browse/HBASE-23855 > Project: HBase > Issue Type: Improvement >Affects Versions: master >Reporter: Zhuoyue Huang >Assignee: Zhuoyue Huang >Priority: Trivial > Fix For: 3.0.0, 2.3.0 > > Attachments: image-2020-02-16-19-23-39-432.png, > image-2020-02-16-19-28-23-449.png > > > I found that the “BytesBufferAllocator Status” in RegionServer Web UI still > using "Bytes" as a fixture unit. I think we should use "MB" or "GB" when the > size is too large !image-2020-02-16-19-23-39-432.png! > > The Web UI after improvement: > !image-2020-02-16-19-28-23-449.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23855) Change bytes size to human readable size for Server Metrics of RegionServer Web UI
[ https://issues.apache.org/jira/browse/HBASE-23855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-23855: --- Fix Version/s: (was: master) 2.3.0 3.0.0 > Change bytes size to human readable size for Server Metrics of RegionServer > Web UI > -- > > Key: HBASE-23855 > URL: https://issues.apache.org/jira/browse/HBASE-23855 > Project: HBase > Issue Type: Improvement >Affects Versions: master >Reporter: Zhuoyue Huang >Assignee: Zhuoyue Huang >Priority: Trivial > Fix For: 3.0.0, 2.3.0 > > Attachments: image-2020-02-16-19-23-39-432.png, > image-2020-02-16-19-28-23-449.png > > > I found that the “BytesBufferAllocator Status” in RegionServer Web UI still > using "Bytes" as a fixture unit. I think we should use "MB" or "GB" when the > size is too large !image-2020-02-16-19-23-39-432.png! > > The Web UI after improvement: > !image-2020-02-16-19-28-23-449.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23822) Fix typo in procedures.jsp
[ https://issues.apache.org/jira/browse/HBASE-23822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang resolved HBASE-23822. Resolution: Fixed Pushed to master, branch-2 and branch-2.2. Thanks [~GeorryHuang] for contributing. > Fix typo in procedures.jsp > -- > > Key: HBASE-23822 > URL: https://issues.apache.org/jira/browse/HBASE-23822 > Project: HBase > Issue Type: Improvement > Components: website >Affects Versions: master >Reporter: Zhuoyue Huang >Assignee: Zhuoyue Huang >Priority: Trivial > Fix For: master > > Attachments: procedures.jsp > > > I think there's a typo.In > hbase-server/src/main/resources/hbase-webapps/master/procedures.jsp. The > Sentences "We do not list Procedures that have completed SUCCESSfully; their > number makes it hard to spot the problematics." are misspelled and need to be > corrected as "We do not list procedures that have completed successfully; > their number makes it hard to spot the problematics." -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23822) Fix typo in procedures.jsp
[ https://issues.apache.org/jira/browse/HBASE-23822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-23822: --- Fix Version/s: (was: master) 2.2.4 2.3.0 3.0.0 > Fix typo in procedures.jsp > -- > > Key: HBASE-23822 > URL: https://issues.apache.org/jira/browse/HBASE-23822 > Project: HBase > Issue Type: Improvement > Components: website >Affects Versions: master >Reporter: Zhuoyue Huang >Assignee: Zhuoyue Huang >Priority: Trivial > Fix For: 3.0.0, 2.3.0, 2.2.4 > > Attachments: procedures.jsp > > > I think there's a typo.In > hbase-server/src/main/resources/hbase-webapps/master/procedures.jsp. The > Sentences "We do not list Procedures that have completed SUCCESSfully; their > number makes it hard to spot the problematics." are misspelled and need to be > corrected as "We do not list procedures that have completed successfully; > their number makes it hard to spot the problematics." -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23822) Fix typo in procedures.jsp
[ https://issues.apache.org/jira/browse/HBASE-23822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-23822: --- Issue Type: Improvement (was: Bug) > Fix typo in procedures.jsp > -- > > Key: HBASE-23822 > URL: https://issues.apache.org/jira/browse/HBASE-23822 > Project: HBase > Issue Type: Improvement > Components: website >Affects Versions: master >Reporter: Zhuoyue Huang >Assignee: Zhuoyue Huang >Priority: Trivial > Fix For: master > > Attachments: procedures.jsp > > > I think there's a typo.In > hbase-server/src/main/resources/hbase-webapps/master/procedures.jsp. The > Sentences "We do not list Procedures that have completed SUCCESSfully; their > number makes it hard to spot the problematics." are misspelled and need to be > corrected as "We do not list procedures that have completed successfully; > their number makes it hard to spot the problematics." -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-23822) Fix typo in procedures.jsp
[ https://issues.apache.org/jira/browse/HBASE-23822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang reassigned HBASE-23822: -- Assignee: Zhuoyue Huang > Fix typo in procedures.jsp > -- > > Key: HBASE-23822 > URL: https://issues.apache.org/jira/browse/HBASE-23822 > Project: HBase > Issue Type: Bug > Components: website >Affects Versions: master >Reporter: Zhuoyue Huang >Assignee: Zhuoyue Huang >Priority: Trivial > Fix For: master > > Attachments: procedures.jsp > > > I think there's a typo.In > hbase-server/src/main/resources/hbase-webapps/master/procedures.jsp. The > Sentences "We do not list Procedures that have completed SUCCESSfully; their > number makes it hard to spot the problematics." are misspelled and need to be > corrected as "We do not list procedures that have completed successfully; > their number makes it hard to spot the problematics." -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-21110) Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()`
[
https://issues.apache.org/jira/browse/HBASE-21110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028313#comment-17028313
]
Guanghao Zhang commented on HBASE-21110:
Port the spark fix here.
> Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()`
> --
>
> Key: HBASE-21110
> URL: https://issues.apache.org/jira/browse/HBASE-21110
> Project: HBase
> Issue Type: Sub-task
>Reporter: Mike Drob
>Assignee: Guanghao Zhang
>Priority: Major
> Labels: jdk11
> Fix For: 3.0.0
>
>
> Using Java 11 RC 1, I get the following warning, probably need to add the
> suggested flag to our scripts?
> {noformat}
> mdrob@mdrob-MBP:~/IdeaProjects/hbase$ java -version
> java version "11" 2018-09-25
> Java(TM) SE Runtime Environment 18.9 (build 11+28)
> Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11+28, mixed mode)
> mdrob@mdrob-MBP:~/IdeaProjects/hbase$ bin/start-hbase.sh
> mdrob@mdrob-MBP:~/IdeaProjects/hbase$ cat
> /Users/mdrob/IdeaProjects/hbase/bin/../logs/hbase-mdrob-master-mdrob-MBP.local.out
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.hadoop.hbase.util.UnsafeAvailChecker
> (file:/Users/mdrob/IdeaProjects/hbase/hbase-common/target/hbase-common-3.0.0-SNAPSHOT.jar)
> to method java.nio.Bits.unaligned()
> WARNING: Please consider reporting this to the maintainers of
> org.apache.hadoop.hbase.util.UnsafeAvailChecker
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-21110) Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()`
[
https://issues.apache.org/jira/browse/HBASE-21110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang reassigned HBASE-21110:
--
Assignee: Guanghao Zhang
> Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()`
> --
>
> Key: HBASE-21110
> URL: https://issues.apache.org/jira/browse/HBASE-21110
> Project: HBase
> Issue Type: Sub-task
>Reporter: Mike Drob
>Assignee: Guanghao Zhang
>Priority: Major
> Labels: jdk11
> Fix For: 3.0.0
>
>
> Using Java 11 RC 1, I get the following warning, probably need to add the
> suggested flag to our scripts?
> {noformat}
> mdrob@mdrob-MBP:~/IdeaProjects/hbase$ java -version
> java version "11" 2018-09-25
> Java(TM) SE Runtime Environment 18.9 (build 11+28)
> Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11+28, mixed mode)
> mdrob@mdrob-MBP:~/IdeaProjects/hbase$ bin/start-hbase.sh
> mdrob@mdrob-MBP:~/IdeaProjects/hbase$ cat
> /Users/mdrob/IdeaProjects/hbase/bin/../logs/hbase-mdrob-master-mdrob-MBP.local.out
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.hadoop.hbase.util.UnsafeAvailChecker
> (file:/Users/mdrob/IdeaProjects/hbase/hbase-common/target/hbase-common-3.0.0-SNAPSHOT.jar)
> to method java.nio.Bits.unaligned()
> WARNING: Please consider reporting this to the maintainers of
> org.apache.hadoop.hbase.util.UnsafeAvailChecker
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-21110) Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()`
[
https://issues.apache.org/jira/browse/HBASE-21110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-21110:
---
Summary: Remove `Illegal reflective access` warning for
`java.nio.Bits.unaligned()` (was: Issues with Unsafe and JDK 11)
> Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()`
> --
>
> Key: HBASE-21110
> URL: https://issues.apache.org/jira/browse/HBASE-21110
> Project: HBase
> Issue Type: Sub-task
>Reporter: Mike Drob
>Priority: Major
> Labels: jdk11
> Fix For: 3.0.0
>
>
> Using Java 11 RC 1, I get the following warning, probably need to add the
> suggested flag to our scripts?
> {noformat}
> mdrob@mdrob-MBP:~/IdeaProjects/hbase$ java -version
> java version "11" 2018-09-25
> Java(TM) SE Runtime Environment 18.9 (build 11+28)
> Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11+28, mixed mode)
> mdrob@mdrob-MBP:~/IdeaProjects/hbase$ bin/start-hbase.sh
> mdrob@mdrob-MBP:~/IdeaProjects/hbase$ cat
> /Users/mdrob/IdeaProjects/hbase/bin/../logs/hbase-mdrob-master-mdrob-MBP.local.out
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.hadoop.hbase.util.UnsafeAvailChecker
> (file:/Users/mdrob/IdeaProjects/hbase/hbase-common/target/hbase-common-3.0.0-SNAPSHOT.jar)
> to method java.nio.Bits.unaligned()
> WARNING: Please consider reporting this to the maintainers of
> org.apache.hadoop.hbase.util.UnsafeAvailChecker
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22972) [JDK11] Support JDK11 LTS in HBase
[ https://issues.apache.org/jira/browse/HBASE-22972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028312#comment-17028312 ] Guanghao Zhang commented on HBASE-22972: When run hbase with hadoop 2.8.5 and jdk11, another warning is "WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/home/xiaohao/code/open_source/hbase/hbase-assembly/target/hbase-3.0.0-SNAPSHOT/lib/hadoop-auth-2.8.5.jar) to method sun.security.krb5.Config.getInstance()". May only support hadoop 3.x when run with jdk11. > [JDK11] Support JDK11 LTS in HBase > -- > > Key: HBASE-22972 > URL: https://issues.apache.org/jira/browse/HBASE-22972 > Project: HBase > Issue Type: Umbrella >Reporter: Duo Zhang >Priority: Blocker > Labels: jdk11 > Fix For: 3.0.0, 2.3.0 > > > This is an umbrella issue for tracking all the problems for JDK11 in HBase. > And we also rely on hadoop. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-22972) [JDK11] Support JDK11 LTS in HBase
[ https://issues.apache.org/jira/browse/HBASE-22972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028262#comment-17028262 ] Guanghao Zhang edited comment on HBASE-22972 at 2/2/20 3:29 AM: The "ungligned" is a method in jdk8 but it is a filed since jdk9. And it seems have different name for jdk9/10 and jdk11/12. was (Author: zghaobac): The ungligned is a method in jdk8 but it is a filed since jdk9. And it seems have different name for jdk9/10 and jdk11/12. > [JDK11] Support JDK11 LTS in HBase > -- > > Key: HBASE-22972 > URL: https://issues.apache.org/jira/browse/HBASE-22972 > Project: HBase > Issue Type: Umbrella >Reporter: Duo Zhang >Priority: Blocker > Labels: jdk11 > Fix For: 3.0.0, 2.3.0 > > > This is an umbrella issue for tracking all the problems for JDK11 in HBase. > And we also rely on hadoop. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22972) [JDK11] Support JDK11 LTS in HBase
[ https://issues.apache.org/jira/browse/HBASE-22972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028262#comment-17028262 ] Guanghao Zhang commented on HBASE-22972: The ungligned is a method in jdk8 but it is a filed since jdk9. And it seems have different name for jdk9/10 and jdk11/12. > [JDK11] Support JDK11 LTS in HBase > -- > > Key: HBASE-22972 > URL: https://issues.apache.org/jira/browse/HBASE-22972 > Project: HBase > Issue Type: Umbrella >Reporter: Duo Zhang >Priority: Blocker > Labels: jdk11 > Fix For: 3.0.0, 2.3.0 > > > This is an umbrella issue for tracking all the problems for JDK11 in HBase. > And we also rely on hadoop. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22972) [JDK11] Support JDK11 LTS in HBase
[ https://issues.apache.org/jira/browse/HBASE-22972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028255#comment-17028255 ] Guanghao Zhang commented on HBASE-22972: The waring message is "WARNING: Illegal reflective access by org.apache.hadoop.hbase.util.UnsafeAvailChecker (file:/home/zhangguanghao1/code/hbase/hbase-common/target/hbase-common-2.3.0-SNAPSHOT.jar) to method java.nio.Bits.unaligned()". And spark had smae problem and resolved by https://issues.apache.org/jira/browse/SPARK-27981 > [JDK11] Support JDK11 LTS in HBase > -- > > Key: HBASE-22972 > URL: https://issues.apache.org/jira/browse/HBASE-22972 > Project: HBase > Issue Type: Umbrella >Reporter: Duo Zhang >Priority: Blocker > Labels: jdk11 > Fix For: 3.0.0, 2.3.0 > > > This is an umbrella issue for tracking all the problems for JDK11 in HBase. > And we also rely on hadoop. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-22972) [JDK11] Support JDK11 LTS in HBase
[ https://issues.apache.org/jira/browse/HBASE-22972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028254#comment-17028254 ] Guanghao Zhang edited comment on HBASE-22972 at 2/2/20 3:02 AM: The runtime means what? I build a package of branch-2 and run with jdk11. There are some warnings but it worked now. I used PE to write/read 1M rows and it worked. The hadoop version is 2.8.5. was (Author: zghaobac): The runtime means what? I build a package of branch-2 and run with jdk11. There are some warnings but it worked now. I used PE to write/read 1M rows and it worked. > [JDK11] Support JDK11 LTS in HBase > -- > > Key: HBASE-22972 > URL: https://issues.apache.org/jira/browse/HBASE-22972 > Project: HBase > Issue Type: Umbrella >Reporter: Duo Zhang >Priority: Blocker > Labels: jdk11 > Fix For: 3.0.0, 2.3.0 > > > This is an umbrella issue for tracking all the problems for JDK11 in HBase. > And we also rely on hadoop. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22972) [JDK11] Support JDK11 LTS in HBase
[ https://issues.apache.org/jira/browse/HBASE-22972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028254#comment-17028254 ] Guanghao Zhang commented on HBASE-22972: The runtime means what? I build a package of branch-2 and run with jdk11. There are some warnings but it worked now. I used PE to write/read 1M rows and it worked. > [JDK11] Support JDK11 LTS in HBase > -- > > Key: HBASE-22972 > URL: https://issues.apache.org/jira/browse/HBASE-22972 > Project: HBase > Issue Type: Umbrella >Reporter: Duo Zhang >Priority: Blocker > Labels: jdk11 > Fix For: 3.0.0, 2.3.0 > > > This is an umbrella issue for tracking all the problems for JDK11 in HBase. > And we also rely on hadoop. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23633) Find a way to handle the corrupt recovered hfiles
[ https://issues.apache.org/jira/browse/HBASE-23633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028252#comment-17028252 ] Guanghao Zhang commented on HBASE-23633: [~pankajkumar] Are you working for this now? > Find a way to handle the corrupt recovered hfiles > - > > Key: HBASE-23633 > URL: https://issues.apache.org/jira/browse/HBASE-23633 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Priority: Major > > Copy the comment from PR review. > > If the file is a corrupt HFile, an exception will be thrown here, which will > cause the region to fail to open. > Maybe we can add a new parameter to control whether to skip the exception, > similar to recover edits which has a parameter > "hbase.hregion.edits.replay.skip.errors"; > > Regions that can't be opened because of detached References or corrupt hfiles > are a fact-of-life. We need work on this issue. This will be a new variant on > the problem -- i.e. bad recovered hfiles. > On adding a config to ignore bad files and just open, thats a bit dangerous > as per @infraio as it could mean silent data loss. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23634) Enable "Split WAL to HFile" by default
[ https://issues.apache.org/jira/browse/HBASE-23634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-23634: --- Parent: HBASE-23286 Issue Type: Sub-task (was: Improvement) > Enable "Split WAL to HFile" by default > -- > > Key: HBASE-23634 > URL: https://issues.apache.org/jira/browse/HBASE-23634 > Project: HBase > Issue Type: Sub-task >Affects Versions: 3.0.0, 2.3.0 >Reporter: Guanghao Zhang >Priority: Blocker > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23633) Find a way to handle the corrupt recovered hfiles
[ https://issues.apache.org/jira/browse/HBASE-23633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-23633: --- Parent: HBASE-23286 Issue Type: Sub-task (was: Improvement) > Find a way to handle the corrupt recovered hfiles > - > > Key: HBASE-23633 > URL: https://issues.apache.org/jira/browse/HBASE-23633 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Priority: Major > > Copy the comment from PR review. > > If the file is a corrupt HFile, an exception will be thrown here, which will > cause the region to fail to open. > Maybe we can add a new parameter to control whether to skip the exception, > similar to recover edits which has a parameter > "hbase.hregion.edits.replay.skip.errors"; > > Regions that can't be opened because of detached References or corrupt hfiles > are a fact-of-life. We need work on this issue. This will be a new variant on > the problem -- i.e. bad recovered hfiles. > On adding a config to ignore bad files and just open, thats a bit dangerous > as per @infraio as it could mean silent data loss. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23741) Data loss when WAL split to HFile enabled
[ https://issues.apache.org/jira/browse/HBASE-23741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024202#comment-17024202 ] Guanghao Zhang commented on HBASE-23741: Sorry, I didn't run ITBLL so there may some case not handle. Will make this feature stable before we release 2.3.0. Thanks for the test. > Data loss when WAL split to HFile enabled > - > > Key: HBASE-23741 > URL: https://issues.apache.org/jira/browse/HBASE-23741 > Project: HBase > Issue Type: Sub-task > Components: MTTR >Affects Versions: 3.0.0, 2.3.0 >Reporter: Pankaj Kumar >Priority: Critical > > Very simple steps as below, > 1. Create table with 1 region > 2. Insert 1 record > 3. Flush the table > 4. Scan table and observe timestamp of the inserted row > 5. Insert same row key with same timestamp as previously inserted but with > different value > 6. Kill -9 RS where table region is online > 7. Start RS > Scan the table and check the result, latest cell must be returned. > Thanks [~sreenivasulureddy] for finding this issue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23739) BoundedRecoveredHFilesOutputSink should read the table descriptor directly
Guanghao Zhang created HBASE-23739: -- Summary: BoundedRecoveredHFilesOutputSink should read the table descriptor directly Key: HBASE-23739 URL: https://issues.apache.org/jira/browse/HBASE-23739 Project: HBase Issue Type: Sub-task Reporter: Guanghao Zhang Read from meta or filesystem? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23286) Improve MTTR: Split WAL to HFile
[
https://issues.apache.org/jira/browse/HBASE-23286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024056#comment-17024056
]
Guanghao Zhang commented on HBASE-23286:
{quote}how BoundedRecoveredHFilesOutputSink will retrieve the table descriptor
when HMaster hasn't finished initialization
{quote}
Good question. This should be a problem. Let me open a new issue for this.
> Improve MTTR: Split WAL to HFile
>
>
> Key: HBASE-23286
> URL: https://issues.apache.org/jira/browse/HBASE-23286
> Project: HBase
> Issue Type: Improvement
> Components: MTTR
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
>
> After HBASE-20724, the compaction event marker is not used anymore when
> failover. So our new proposal is split WAL to HFile to imporve MTTR. It has 3
> steps:
> # Read WAL and write HFile to region’s column family’s recovered.hfiles
> directory.
> # Open region.
> # Bulkload the recovered.hfiles for every column family.
> The design doc was attathed by a google doc. Any suggestions are welcomed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23156) start-hbase.sh failed with ClassNotFoundException when build with hadoop3
[
https://issues.apache.org/jira/browse/HBASE-23156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-23156:
---
Release Note: Introduce a new
hbase-assembly/src/main/assembly/hadoop-three-compat.xml for build with hadoop
3.x.
> start-hbase.sh failed with ClassNotFoundException when build with hadoop3
> -
>
> Key: HBASE-23156
> URL: https://issues.apache.org/jira/browse/HBASE-23156
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
>
> {code:java}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/ctc/wstx/io/InputBootstrapperException in thread "main"
> java.lang.NoClassDefFoundError: com/ctc/wstx/io/InputBootstrapper at
> org.apache.hadoop.hbase.util.HBaseConfTool.main(HBaseConfTool.java:39)Caused
> by: java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper at
> java.net.URLClassLoader.findClass(URLClassLoader.java:382) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 1 moreException in
> thread "main" java.lang.NoClassDefFoundError:
> com/ctc/wstx/io/InputBootstrapper at
> org.apache.hadoop.hbase.zookeeper.ZKServerTool.main(ZKServerTool.java:63)Caused
> by: java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper at
> java.net.URLClassLoader.findClass(URLClassLoader.java:382) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 1 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-23156) start-hbase.sh failed with ClassNotFoundException when build with hadoop3
[
https://issues.apache.org/jira/browse/HBASE-23156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang resolved HBASE-23156.
Resolution: Fixed
Pushed to branch-2 and master. Thanks [~zhangduo] for reviewing.
> start-hbase.sh failed with ClassNotFoundException when build with hadoop3
> -
>
> Key: HBASE-23156
> URL: https://issues.apache.org/jira/browse/HBASE-23156
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
>
> {code:java}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/ctc/wstx/io/InputBootstrapperException in thread "main"
> java.lang.NoClassDefFoundError: com/ctc/wstx/io/InputBootstrapper at
> org.apache.hadoop.hbase.util.HBaseConfTool.main(HBaseConfTool.java:39)Caused
> by: java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper at
> java.net.URLClassLoader.findClass(URLClassLoader.java:382) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 1 moreException in
> thread "main" java.lang.NoClassDefFoundError:
> com/ctc/wstx/io/InputBootstrapper at
> org.apache.hadoop.hbase.zookeeper.ZKServerTool.main(ZKServerTool.java:63)Caused
> by: java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper at
> java.net.URLClassLoader.findClass(URLClassLoader.java:382) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 1 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23347) Pluggable RPC authentication
[ https://issues.apache.org/jira/browse/HBASE-23347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019203#comment-17019203 ] Guanghao Zhang commented on HBASE-23347: TestThriftSpnegoHttpFallbackServer and TestSecureRESTServer failed. May releated to this issue. > Pluggable RPC authentication > > > Key: HBASE-23347 > URL: https://issues.apache.org/jira/browse/HBASE-23347 > Project: HBase > Issue Type: Improvement > Components: rpc, security >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > Today in HBase, we rely on SASL to implement Kerberos and delegation token > authentication. The RPC client and server logic is very tightly coupled to > our three authentication mechanism (the previously two mentioned plus simple > auth'n) for no good reason (other than "that's how it was built", best as I > can tell). > SASL's function is to decouple the "application" from how a request is being > authenticated, which means that, to support a variety of other authentication > approaches, we just need to be a little more flexible in letting developers > create their own authentication mechanism for HBase. > This is less for the "average joe" user to write their own authentication > plugin (eek), but more to allow us HBase developers to start iterating, see > what is possible. > I'll attach a full write-up on what I have today as to how I think we can add > these abstractions, as well as an initial implementation of this idea, with a > unit test that shows an end-to-end authentication solution against HBase. > cc/ [~wchevreuil] as he's been working with me behind the scenes, giving lots > of great feedback and support. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23156) start-hbase.sh failed with ClassNotFoundException when build with hadoop3
[
https://issues.apache.org/jira/browse/HBASE-23156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17018850#comment-17018850
]
Guanghao Zhang commented on HBASE-23156:
And there is one another problem. Need to config
hbase.unsafe.stream.capability.enforce to false when start standalone hbase.
Because the LocalFileSystem not support hflush and hsync, the
CommonFSUtils.hasCapability will fail...
> start-hbase.sh failed with ClassNotFoundException when build with hadoop3
> -
>
> Key: HBASE-23156
> URL: https://issues.apache.org/jira/browse/HBASE-23156
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
>
> {code:java}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/ctc/wstx/io/InputBootstrapperException in thread "main"
> java.lang.NoClassDefFoundError: com/ctc/wstx/io/InputBootstrapper at
> org.apache.hadoop.hbase.util.HBaseConfTool.main(HBaseConfTool.java:39)Caused
> by: java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper at
> java.net.URLClassLoader.findClass(URLClassLoader.java:382) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 1 moreException in
> thread "main" java.lang.NoClassDefFoundError:
> com/ctc/wstx/io/InputBootstrapper at
> org.apache.hadoop.hbase.zookeeper.ZKServerTool.main(ZKServerTool.java:63)Caused
> by: java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper at
> java.net.URLClassLoader.findClass(URLClassLoader.java:382) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 1 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-23693) Split failure may cause region hole and data loss when use zk assign
[
https://issues.apache.org/jira/browse/HBASE-23693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang reassigned HBASE-23693:
--
Assignee: tianhang tang
> Split failure may cause region hole and data loss when use zk assign
>
>
> Key: HBASE-23693
> URL: https://issues.apache.org/jira/browse/HBASE-23693
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 1.4.8
>Reporter: tianhang tang
>Assignee: tianhang tang
>Priority: Critical
> Attachments: HBASE-23693.branch-1.001.patch
>
>
> to mock this case, I add a sleep code in SplitTransactionImpl.excute after
> the PONR and before openDaughters:
> {code:java}
> public PairOfSameType execute(final Server server,
> final RegionServerServices services, User user) throws IOException {
> this.server = server;
> this.rsServices = services;
> useZKForAssignment = server == null ? true :
> ConfigUtil.useZKForAssignment(server.getConfiguration());
> if (useCoordinatedStateManager(server)) {
> std =
> ((BaseCoordinatedStateManager) server.getCoordinatedStateManager())
> .getSplitTransactionCoordination().getDefaultDetails();
> }
> PairOfSameType regions = createDaughters(server, services, user);
> if (this.parent.getCoprocessorHost() != null) {
> if (user == null) {
> parent.getCoprocessorHost().preSplitAfterPONR();
> } else {
> try {
> user.getUGI().doAs(new PrivilegedExceptionAction() {
> @Override
> public Void run() throws Exception {
> parent.getCoprocessorHost().preSplitAfterPONR();
> return null;
> }
> });
> } catch (InterruptedException ie) {
> InterruptedIOException iioe = new InterruptedIOException();
> iioe.initCause(ie);
> throw iioe;
> }
> }
> }
>
> //sleep here!!!
> try {
> Thread.sleep(1000 * 60 * 60);
> } catch (InterruptedException e) {
> e.printStackTrace();
> }
> regions = stepsAfterPONR(server, services, regions, user);
> transition(SplitTransactionPhase.COMPLETED);
> return regions;
> }
> {code}
> so the split transaction will hang.
> then i try to reproduce this problem:
> 1.Create a test table and move it into a test rsgroup, there is only 1 RS in
> the test group
> 2.Trigger a region split
> 3.The split transaction step after the PONR and sleep, regioninfo in meta has
> been updated
> 4.Kill the RS process to mock machine crash
> 5.ServerCrashProcedure cleanup SPLITING_NEW region, the daughter regions will
> be deleted
> 6.ServerCrashProcedure try to assign the parent region, because RS is down
> and assign fails, the region status is set to FAILED_OPEN and put back into
> regionsInTransition. But at this time, due to RS crash, the node of the
> region under ZK region-in-transition no longer exist
> 7.CatalogJanitor thread is blocked due to RIT
> 8.Switch active master
> 9.The CatalogJanitor thread on the new master executes normally and the
> parent region is cleaned up because split = true && offline = true in the
> meta table
> 10.We have a hole in the test table and loss data.
>
> I modified the code when ServerCrashProcedure cleans up the child regions, it
> will update the parent regioninfo in the meta table, and this problem is no
> longer reproduced.
> I will upload the patch later.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23156) start-hbase.sh failed with ClassNotFoundException when build with hadoop3
[
https://issues.apache.org/jira/browse/HBASE-23156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17018778#comment-17018778
]
Guanghao Zhang commented on HBASE-23156:
Failed since HBASE-22264, the diffenence is remove two jar from the exclude set:
com.fasterxml.woodstox:woodstox-core
org.codehaus.woodstox:stax2-api
> start-hbase.sh failed with ClassNotFoundException when build with hadoop3
> -
>
> Key: HBASE-23156
> URL: https://issues.apache.org/jira/browse/HBASE-23156
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
>
> {code:java}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/ctc/wstx/io/InputBootstrapperException in thread "main"
> java.lang.NoClassDefFoundError: com/ctc/wstx/io/InputBootstrapper at
> org.apache.hadoop.hbase.util.HBaseConfTool.main(HBaseConfTool.java:39)Caused
> by: java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper at
> java.net.URLClassLoader.findClass(URLClassLoader.java:382) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 1 moreException in
> thread "main" java.lang.NoClassDefFoundError:
> com/ctc/wstx/io/InputBootstrapper at
> org.apache.hadoop.hbase.zookeeper.ZKServerTool.main(ZKServerTool.java:63)Caused
> by: java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper at
> java.net.URLClassLoader.findClass(URLClassLoader.java:382) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 1 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HBASE-23156) start-hbase.sh failed with ClassNotFoundException when build with hadoop3
[
https://issues.apache.org/jira/browse/HBASE-23156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-23156 started by Guanghao Zhang.
--
> start-hbase.sh failed with ClassNotFoundException when build with hadoop3
> -
>
> Key: HBASE-23156
> URL: https://issues.apache.org/jira/browse/HBASE-23156
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
>
> {code:java}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/ctc/wstx/io/InputBootstrapperException in thread "main"
> java.lang.NoClassDefFoundError: com/ctc/wstx/io/InputBootstrapper at
> org.apache.hadoop.hbase.util.HBaseConfTool.main(HBaseConfTool.java:39)Caused
> by: java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper at
> java.net.URLClassLoader.findClass(URLClassLoader.java:382) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 1 moreException in
> thread "main" java.lang.NoClassDefFoundError:
> com/ctc/wstx/io/InputBootstrapper at
> org.apache.hadoop.hbase.zookeeper.ZKServerTool.main(ZKServerTool.java:63)Caused
> by: java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper at
> java.net.URLClassLoader.findClass(URLClassLoader.java:382) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 1 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23156) start-hbase.sh failed with ClassNotFoundException when build with hadoop3
[
https://issues.apache.org/jira/browse/HBASE-23156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-23156:
---
Affects Version/s: 2.3.0
3.0.0
> start-hbase.sh failed with ClassNotFoundException when build with hadoop3
> -
>
> Key: HBASE-23156
> URL: https://issues.apache.org/jira/browse/HBASE-23156
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
>
> {code:java}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/ctc/wstx/io/InputBootstrapperException in thread "main"
> java.lang.NoClassDefFoundError: com/ctc/wstx/io/InputBootstrapper at
> org.apache.hadoop.hbase.util.HBaseConfTool.main(HBaseConfTool.java:39)Caused
> by: java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper at
> java.net.URLClassLoader.findClass(URLClassLoader.java:382) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 1 moreException in
> thread "main" java.lang.NoClassDefFoundError:
> com/ctc/wstx/io/InputBootstrapper at
> org.apache.hadoop.hbase.zookeeper.ZKServerTool.main(ZKServerTool.java:63)Caused
> by: java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper at
> java.net.URLClassLoader.findClass(URLClassLoader.java:382) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 1 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HBASE-23156) start-hbase.sh failed with ClassNotFoundException when build with hadoop3
[
https://issues.apache.org/jira/browse/HBASE-23156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang reassigned HBASE-23156:
--
Assignee: Guanghao Zhang
> start-hbase.sh failed with ClassNotFoundException when build with hadoop3
> -
>
> Key: HBASE-23156
> URL: https://issues.apache.org/jira/browse/HBASE-23156
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
>
> {code:java}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/ctc/wstx/io/InputBootstrapperException in thread "main"
> java.lang.NoClassDefFoundError: com/ctc/wstx/io/InputBootstrapper at
> org.apache.hadoop.hbase.util.HBaseConfTool.main(HBaseConfTool.java:39)Caused
> by: java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper at
> java.net.URLClassLoader.findClass(URLClassLoader.java:382) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 1 moreException in
> thread "main" java.lang.NoClassDefFoundError:
> com/ctc/wstx/io/InputBootstrapper at
> org.apache.hadoop.hbase.zookeeper.ZKServerTool.main(ZKServerTool.java:63)Caused
> by: java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper at
> java.net.URLClassLoader.findClass(URLClassLoader.java:382) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 1 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-23547) Release 2.2.3
[ https://issues.apache.org/jira/browse/HBASE-23547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang resolved HBASE-23547. Assignee: Guanghao Zhang Resolution: Fixed > Release 2.2.3 > - > > Key: HBASE-23547 > URL: https://issues.apache.org/jira/browse/HBASE-23547 > Project: HBase > Issue Type: Umbrella >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23691) Add 2.2.3 to download page
[ https://issues.apache.org/jira/browse/HBASE-23691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang resolved HBASE-23691. Assignee: Guanghao Zhang Resolution: Fixed Pushed to master. Thanks [~janh] for reviewing. > Add 2.2.3 to download page > -- > > Key: HBASE-23691 > URL: https://issues.apache.org/jira/browse/HBASE-23691 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23691) Add 2.2.3 to download page
[ https://issues.apache.org/jira/browse/HBASE-23691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-23691: --- Fix Version/s: 3.0.0 > Add 2.2.3 to download page > -- > > Key: HBASE-23691 > URL: https://issues.apache.org/jira/browse/HBASE-23691 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23692) Set version as 2.2.4-SNAPSHOT in branch-2.2
[ https://issues.apache.org/jira/browse/HBASE-23692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang resolved HBASE-23692. Fix Version/s: 2.2.4 Assignee: Guanghao Zhang Resolution: Fixed > Set version as 2.2.4-SNAPSHOT in branch-2.2 > --- > > Key: HBASE-23692 > URL: https://issues.apache.org/jira/browse/HBASE-23692 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 2.2.4 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23638) Set version to 2.2.3 in branch-2.2 for first RC of 2.2.3
[ https://issues.apache.org/jira/browse/HBASE-23638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang resolved HBASE-23638. Assignee: Guanghao Zhang Resolution: Fixed > Set version to 2.2.3 in branch-2.2 for first RC of 2.2.3 > > > Key: HBASE-23638 > URL: https://issues.apache.org/jira/browse/HBASE-23638 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 2.2.3 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23637) Generate CHANGES.md and RELEASENOTES.md for 2.2.3
[ https://issues.apache.org/jira/browse/HBASE-23637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang resolved HBASE-23637. Assignee: Guanghao Zhang Resolution: Fixed > Generate CHANGES.md and RELEASENOTES.md for 2.2.3 > - > > Key: HBASE-23637 > URL: https://issues.apache.org/jira/browse/HBASE-23637 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 2.2.3 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23692) Set version as 2.2.4-SNAPSHOT in branch-2.2
Guanghao Zhang created HBASE-23692: -- Summary: Set version as 2.2.4-SNAPSHOT in branch-2.2 Key: HBASE-23692 URL: https://issues.apache.org/jira/browse/HBASE-23692 Project: HBase Issue Type: Sub-task Reporter: Guanghao Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23691) Add 2.2.3 to download page
Guanghao Zhang created HBASE-23691: -- Summary: Add 2.2.3 to download page Key: HBASE-23691 URL: https://issues.apache.org/jira/browse/HBASE-23691 Project: HBase Issue Type: Sub-task Reporter: Guanghao Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23684) NPE HFilesOutputSink
[
https://issues.apache.org/jira/browse/HBASE-23684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015558#comment-17015558
]
Guanghao Zhang commented on HBASE-23684:
tableDesc or tableName is null. [~stack] How many tables in your test cluster?
> NPE HFilesOutputSink
>
>
> Key: HBASE-23684
> URL: https://issues.apache.org/jira/browse/HBASE-23684
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.3.0
>Reporter: Michael Stack
>Priority: Major
>
> Enabling the new split to hfiles feature, HBASE-23286, running branch-2 tip,
> I see this out on RegionServers:
> {code}
> 2020-01-13 17:37:08,204 INFO org.apache.hadoop.hbase.wal.OutputSink: 3 split
> writer threads finished
> 2020-01-13 17:37:08,233 INFO org.apache.hadoop.hbase.wal.WALSplitter:
> Processed 1007 edits across 0 regions cost 284 ms; edits skipped=76;
> WAL=hdfs://nameservice1/hbase/genie/WALs/hbasedn101.example.org,16020,1578934806382-splitting/hbasedn101.example.org%2C16020%2C1578934806382.1578937008832,
> size=128.5 M, length=134708720, corrupted=false, progress failed=true
> 2020-01-13 17:37:08,234 WARN
> org.apache.hadoop.hbase.regionserver.SplitLogWorker: log splitting of
> WALs/hbasedn101.example.org,16020,1578934806382-splitting/hbasedn101.example.org%2C16020%2C1578934806382.1578937008832
> failed, returning error
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.writeRemainingEntryBuffers(BoundedRecoveredHFilesOutputSink.java:173)
> at
> org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.close(BoundedRecoveredHFilesOutputSink.java:140)
> at
> org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:339)
> at
> org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:181)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.splitLog(SplitLogWorker.java:105)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.lambda$new$0(SplitLogWorker.java:84)
> at
> org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:70)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.configContextForNonMetaWriter(BoundedRecoveredHFilesOutputSink.java:225)
> at
> org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.createRecoveredHFileWriter(BoundedRecoveredHFilesOutputSink.java:213)
> at
> org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.append(BoundedRecoveredHFilesOutputSink.java:117)
> at
> org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.lambda$writeRemainingEntryBuffers$3(BoundedRecoveredHFilesOutputSink.java:155)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> {code}
> It is a bit odd because log says there were zero regions. Not sure what that
> was about.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (HBASE-23662) Replace HColumnDescriptor(String cf) with ColumnFamilyDescriptor
[
https://issues.apache.org/jira/browse/HBASE-23662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang reopened HBASE-23662:
> Replace HColumnDescriptor(String cf) with ColumnFamilyDescriptor
>
>
> Key: HBASE-23662
> URL: https://issues.apache.org/jira/browse/HBASE-23662
> Project: HBase
> Issue Type: Task
>Affects Versions: 3.0.0
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0
>
>
> We have deprecated HColumnDescriptor and HTableDescriptor some time back in
> 2.0.0. Although there are many use-cases still using them, we can start
> replacing them with builder pattern: ColumnFamilyDescriptorBuilder and
> TableDescriptorBuilder.
> In an attempt to do so, starting with replacing specific constructor of
> HColumnDescriptor:
> {code:java}
> public HTableDescriptor(final TableName name) {
> this(new ModifyableTableDescriptor(name));
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23662) Replace HColumnDescriptor(String cf) with ColumnFamilyDescriptor
[
https://issues.apache.org/jira/browse/HBASE-23662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013931#comment-17013931
]
Guanghao Zhang commented on HBASE-23662:
[https://builds.apache.org/job/HBase-Flaky-Tests/job/master/5277/testReport/junit/org.apache.hadoop.hbase.client/TestShell/testRunShellTests/]
> Replace HColumnDescriptor(String cf) with ColumnFamilyDescriptor
>
>
> Key: HBASE-23662
> URL: https://issues.apache.org/jira/browse/HBASE-23662
> Project: HBase
> Issue Type: Task
>Affects Versions: 3.0.0
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0
>
>
> We have deprecated HColumnDescriptor and HTableDescriptor some time back in
> 2.0.0. Although there are many use-cases still using them, we can start
> replacing them with builder pattern: ColumnFamilyDescriptorBuilder and
> TableDescriptorBuilder.
> In an attempt to do so, starting with replacing specific constructor of
> HColumnDescriptor:
> {code:java}
> public HTableDescriptor(final TableName name) {
> this(new ModifyableTableDescriptor(name));
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23662) Replace HColumnDescriptor(String cf) with ColumnFamilyDescriptor
[
https://issues.apache.org/jira/browse/HBASE-23662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013930#comment-17013930
]
Guanghao Zhang commented on HBASE-23662:
All shell unit test failed. Because the code in shell still need
HColumnDescriptor(String cf). Please upload a addendum patch to update shell
use case too. Thanks.
> Replace HColumnDescriptor(String cf) with ColumnFamilyDescriptor
>
>
> Key: HBASE-23662
> URL: https://issues.apache.org/jira/browse/HBASE-23662
> Project: HBase
> Issue Type: Task
>Affects Versions: 3.0.0
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Fix For: 3.0.0
>
>
> We have deprecated HColumnDescriptor and HTableDescriptor some time back in
> 2.0.0. Although there are many use-cases still using them, we can start
> replacing them with builder pattern: ColumnFamilyDescriptorBuilder and
> TableDescriptorBuilder.
> In an attempt to do so, starting with replacing specific constructor of
> HColumnDescriptor:
> {code:java}
> public HTableDescriptor(final TableName name) {
> this(new ModifyableTableDescriptor(name));
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23659) BaseLoadBalancer#wouldLowerAvailability should consider region replicas
[ https://issues.apache.org/jira/browse/HBASE-23659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013929#comment-17013929 ] Guanghao Zhang commented on HBASE-23659: |master.assignment.TestRegionReplicaSplit|20.0% (6 / 30)| > BaseLoadBalancer#wouldLowerAvailability should consider region replicas > --- > > Key: HBASE-23659 > URL: https://issues.apache.org/jira/browse/HBASE-23659 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.3 > > > Found this issue when try to fix the flaky unit test TestRegionReplicaSplit. > It may fail as > java.lang.AssertionError: Splitted regions should not be assigned to same > region server. > See > [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/5227/testReport/junit/org.apache.hadoop.hbase.master.assignment/TestRegionReplicaSplit/testRegionReplicaSplitRegionAssignment/]. > > Now wouldLowerAvailability method only consider the primary regions. The > replica region can't assign to same server with primary region. But can be > assigned to same server with other replica regions. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23658) Fix flaky TestSnapshotFromMaster
[
https://issues.apache.org/jira/browse/HBASE-23658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013928#comment-17013928
]
Guanghao Zhang commented on HBASE-23658:
|master.cleaner.TestSnapshotFromMaster|20.0% (6 / 30)|
Not flaky anymore. See
[https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.2/lastSuccessfulBuild/artifact/dashboard.html].
> Fix flaky TestSnapshotFromMaster
>
>
> Key: HBASE-23658
> URL: https://issues.apache.org/jira/browse/HBASE-23658
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.2.3, 2.1.9
>
> Attachments: HBASE-23658.branch-2.2.001.patch
>
>
> testAsyncSnapshotWillNotBlockSnapshotHFileCleaner is flaky. The assert may
> fail.
> {code:java}
> assertTrue(master.getSnapshotManager().isTakingAnySnapshot());
> future.get(); // in branch-2.2, here is Thread.sleep
> assertFalse(master.getSnapshotManager().isTakingAnySnapshot());
> {code}
> See
> [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/5227/testReport/junit/org.apache.hadoop.hbase.master.cleaner/TestSnapshotFromMaster/testAsyncSnapshotWillNotBlockSnapshotHFileCleaner/]
>
> [https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.2/lastSuccessfulBuild/artifact/dashboard.html]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23655) Fix flaky TestRSGroupsKillRS: should wait the SCP to finish
[ https://issues.apache.org/jira/browse/HBASE-23655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013927#comment-17013927 ] Guanghao Zhang commented on HBASE-23655: |rsgroup.TestRSGroupsKillRS|3.3% (1 / 30)| Not flaky anymore. See [https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.2/lastSuccessfulBuild/artifact/dashboard.html] > Fix flaky TestRSGroupsKillRS: should wait the SCP to finish > --- > > Key: HBASE-23655 > URL: https://issues.apache.org/jira/browse/HBASE-23655 > Project: HBase > Issue Type: Bug >Affects Versions: 2.2.2 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.3 > > > In teardownAfterMethod, it will moveTables and delete RSGroups. It will fail > if not wait SCP to finish. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23601) OutputSink.WriterThread exception gets stuck and repeated indefinietly
[
https://issues.apache.org/jira/browse/HBASE-23601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17012637#comment-17012637
]
Guanghao Zhang commented on HBASE-23601:
I plan to release 2.2.3 now. Let me revert it from branch-2.2 as this not
resolved. Will merge this after release. Thanks.
> OutputSink.WriterThread exception gets stuck and repeated indefinietly
> --
>
> Key: HBASE-23601
> URL: https://issues.apache.org/jira/browse/HBASE-23601
> Project: HBase
> Issue Type: Bug
> Components: read replicas
>Affects Versions: 2.2.2
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.1.9, 2.2.4
>
>
> When a WriterThread runs into an exception (ie: NotServingRegionException),
> the exception is stored in the controller. It is never removed and can not be
> overwritten either.
>
> {code:java}
> public void run() {
> try {
> doRun();
> } catch (Throwable t) {
> LOG.error("Exiting thread", t);
> controller.writerThreadError(t);
> }
> }{code}
> Thanks to this every time PipelineController.checkForErrors() is called the
> same old exception is rethrown.
>
> For example in RegionReplicaReplicationEndpoint.replicate there is a while
> loop that does the actual replicating. Every time it loops, it calls
> checkForErrors(), catches the rethrown exception, logs it but does nothing
> about it. This results in ~2GB log files in ~5min in my experience.
>
> My proposal would be to clean up the stored exception when it reaches
> RegionReplicaReplicationEndpoint.replicate and make sure we restart the
> WriterThread that died throwing it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23666) Backport "HBASE-23660 hbase:meta's table.jsp ref to wrong rs address" to branch-2
[ https://issues.apache.org/jira/browse/HBASE-23666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-23666: --- Fix Version/s: 2.2.3 > Backport "HBASE-23660 hbase:meta's table.jsp ref to wrong rs address" to > branch-2 > - > > Key: HBASE-23666 > URL: https://issues.apache.org/jira/browse/HBASE-23666 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 2.2.2 >Reporter: Lijin Bin >Assignee: Lijin Bin >Priority: Major > Fix For: 2.3.0, 2.2.3 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23666) Backport "HBASE-23660 hbase:meta's table.jsp ref to wrong rs address" to branch-2
[ https://issues.apache.org/jira/browse/HBASE-23666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-23666: --- Fix Version/s: (was: 2.2.3) > Backport "HBASE-23660 hbase:meta's table.jsp ref to wrong rs address" to > branch-2 > - > > Key: HBASE-23666 > URL: https://issues.apache.org/jira/browse/HBASE-23666 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 2.2.2 >Reporter: Lijin Bin >Assignee: Lijin Bin >Priority: Major > Fix For: 2.3.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23651) Region balance throttling can be disabled
[ https://issues.apache.org/jira/browse/HBASE-23651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-23651: --- Fix Version/s: (was: 2.2.4) 2.2.3 > Region balance throttling can be disabled > - > > Key: HBASE-23651 > URL: https://issues.apache.org/jira/browse/HBASE-23651 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.2.3 >Reporter: Lijin Bin >Assignee: Lijin Bin >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.3 > > > HBASE-17178 Add region balance throttling, but it can not be disabled, > sometimes we need no throttle and balance the cluster as fast as possible. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23663) Allow dot and hyphen in Profiler's URL
[
https://issues.apache.org/jira/browse/HBASE-23663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-23663:
---
Fix Version/s: (was: 2.2.4)
2.2.3
> Allow dot and hyphen in Profiler's URL
> --
>
> Key: HBASE-23663
> URL: https://issues.apache.org/jira/browse/HBASE-23663
> Project: HBase
> Issue Type: Bug
> Components: profiler
>Affects Versions: 3.0.0, 2.2.0, 2.3.0, 2.1.5
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Minor
> Fix For: 3.0.0, 2.3.0, 2.2.3, 2.1.9
>
>
> ProfileOutputServlet sanitizes the input URL but fails when the query string
> contains dot or hyphen. These are valid characters for the hostname.
> Example URL part:
> /prof-output/async-prof-pid-122466-cpu-1.svg?host=myhost-1.example.com=16010
>
> In this case the user gets the following error message:
> {noformat}
> HTTP ERROR: 500
> Problem accessing /prof-output/async-prof-pid-122466-cpu-1.svg. Reason:
> java.lang.RuntimeException: Non-alphanumeric data found in input,
> aborting.
> Powered by Jetty:// 9.3.27.v20190418 {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-23658) Fix flaky TestSnapshotFromMaster
[
https://issues.apache.org/jira/browse/HBASE-23658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang resolved HBASE-23658.
Resolution: Fixed
> Fix flaky TestSnapshotFromMaster
>
>
> Key: HBASE-23658
> URL: https://issues.apache.org/jira/browse/HBASE-23658
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.2.3, 2.1.9
>
> Attachments: HBASE-23658.branch-2.2.001.patch
>
>
> testAsyncSnapshotWillNotBlockSnapshotHFileCleaner is flaky. The assert may
> fail.
> {code:java}
> assertTrue(master.getSnapshotManager().isTakingAnySnapshot());
> future.get(); // in branch-2.2, here is Thread.sleep
> assertFalse(master.getSnapshotManager().isTakingAnySnapshot());
> {code}
> See
> [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/5227/testReport/junit/org.apache.hadoop.hbase.master.cleaner/TestSnapshotFromMaster/testAsyncSnapshotWillNotBlockSnapshotHFileCleaner/]
>
> [https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.2/lastSuccessfulBuild/artifact/dashboard.html]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-23659) BaseLoadBalancer#wouldLowerAvailability should consider region replicas
[ https://issues.apache.org/jira/browse/HBASE-23659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang resolved HBASE-23659. Resolution: Fixed > BaseLoadBalancer#wouldLowerAvailability should consider region replicas > --- > > Key: HBASE-23659 > URL: https://issues.apache.org/jira/browse/HBASE-23659 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.3 > > > Found this issue when try to fix the flaky unit test TestRegionReplicaSplit. > It may fail as > java.lang.AssertionError: Splitted regions should not be assigned to same > region server. > See > [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/5227/testReport/junit/org.apache.hadoop.hbase.master.assignment/TestRegionReplicaSplit/testRegionReplicaSplitRegionAssignment/]. > > Now wouldLowerAvailability method only consider the primary regions. The > replica region can't assign to same server with primary region. But can be > assigned to same server with other replica regions. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23655) Fix flaky TestRSGroupsKillRS: should wait the SCP to finish
[ https://issues.apache.org/jira/browse/HBASE-23655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang resolved HBASE-23655. Resolution: Fixed > Fix flaky TestRSGroupsKillRS: should wait the SCP to finish > --- > > Key: HBASE-23655 > URL: https://issues.apache.org/jira/browse/HBASE-23655 > Project: HBase > Issue Type: Bug >Affects Versions: 2.2.2 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.3 > > > In teardownAfterMethod, it will moveTables and delete RSGroups. It will fail > if not wait SCP to finish. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23659) BaseLoadBalancer#wouldLowerAvailability should consider region replicas
[ https://issues.apache.org/jira/browse/HBASE-23659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17012552#comment-17012552 ] Guanghao Zhang commented on HBASE-23659: Pushed to branch-2.2+. Thanks all for reviewing. > BaseLoadBalancer#wouldLowerAvailability should consider region replicas > --- > > Key: HBASE-23659 > URL: https://issues.apache.org/jira/browse/HBASE-23659 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > > Found this issue when try to fix the flaky unit test TestRegionReplicaSplit. > It may fail as > java.lang.AssertionError: Splitted regions should not be assigned to same > region server. > See > [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/5227/testReport/junit/org.apache.hadoop.hbase.master.assignment/TestRegionReplicaSplit/testRegionReplicaSplitRegionAssignment/]. > > Now wouldLowerAvailability method only consider the primary regions. The > replica region can't assign to same server with primary region. But can be > assigned to same server with other replica regions. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23659) BaseLoadBalancer#wouldLowerAvailability should consider region replicas
[ https://issues.apache.org/jira/browse/HBASE-23659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-23659: --- Fix Version/s: 2.2.3 2.3.0 3.0.0 > BaseLoadBalancer#wouldLowerAvailability should consider region replicas > --- > > Key: HBASE-23659 > URL: https://issues.apache.org/jira/browse/HBASE-23659 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.3 > > > Found this issue when try to fix the flaky unit test TestRegionReplicaSplit. > It may fail as > java.lang.AssertionError: Splitted regions should not be assigned to same > region server. > See > [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/5227/testReport/junit/org.apache.hadoop.hbase.master.assignment/TestRegionReplicaSplit/testRegionReplicaSplitRegionAssignment/]. > > Now wouldLowerAvailability method only consider the primary regions. The > replica region can't assign to same server with primary region. But can be > assigned to same server with other replica regions. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23286) Improve MTTR: Split WAL to HFile
[
https://issues.apache.org/jira/browse/HBASE-23286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17012543#comment-17012543
]
Guanghao Zhang commented on HBASE-23286:
{quote}bq. I think [~zghao] has already ported the patch to our internal
branch-2 based branch and tested the feature?
{quote}
Yes. I tested this on our internal branch.
> Improve MTTR: Split WAL to HFile
>
>
> Key: HBASE-23286
> URL: https://issues.apache.org/jira/browse/HBASE-23286
> Project: HBase
> Issue Type: Improvement
> Components: MTTR
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
>
> After HBASE-20724, the compaction event marker is not used anymore when
> failover. So our new proposal is split WAL to HFile to imporve MTTR. It has 3
> steps:
> # Read WAL and write HFile to region’s column family’s recovered.hfiles
> directory.
> # Open region.
> # Bulkload the recovered.hfiles for every column family.
> The design doc was attathed by a google doc. Any suggestions are welcomed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23286) Improve MTTR: Split WAL to HFile
[ https://issues.apache.org/jira/browse/HBASE-23286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17012401#comment-17012401 ] Guanghao Zhang commented on HBASE-23286: [~stack] Let me take a look about this. > Improve MTTR: Split WAL to HFile > > > Key: HBASE-23286 > URL: https://issues.apache.org/jira/browse/HBASE-23286 > Project: HBase > Issue Type: Improvement > Components: MTTR >Affects Versions: 3.0.0, 2.3.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > After HBASE-20724, the compaction event marker is not used anymore when > failover. So our new proposal is split WAL to HFile to imporve MTTR. It has 3 > steps: > # Read WAL and write HFile to region’s column family’s recovered.hfiles > directory. > # Open region. > # Bulkload the recovered.hfiles for every column family. > The design doc was attathed by a google doc. Any suggestions are welcomed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23655) Fix flaky TestRSGroupsKillRS: should wait the SCP to finish
[ https://issues.apache.org/jira/browse/HBASE-23655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011717#comment-17011717 ] Guanghao Zhang commented on HBASE-23655: Pushed to branch-2.2+. > Fix flaky TestRSGroupsKillRS: should wait the SCP to finish > --- > > Key: HBASE-23655 > URL: https://issues.apache.org/jira/browse/HBASE-23655 > Project: HBase > Issue Type: Bug >Affects Versions: 2.2.2 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.3 > > > In teardownAfterMethod, it will moveTables and delete RSGroups. It will fail > if not wait SCP to finish. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-23658) Fix flaky TestSnapshotFromMaster
[
https://issues.apache.org/jira/browse/HBASE-23658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang reassigned HBASE-23658:
--
Assignee: Guanghao Zhang
> Fix flaky TestSnapshotFromMaster
>
>
> Key: HBASE-23658
> URL: https://issues.apache.org/jira/browse/HBASE-23658
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.2.3, 2.1.9
>
> Attachments: HBASE-23658.branch-2.2.001.patch
>
>
> testAsyncSnapshotWillNotBlockSnapshotHFileCleaner is flaky. The assert may
> fail.
> {code:java}
> assertTrue(master.getSnapshotManager().isTakingAnySnapshot());
> future.get(); // in branch-2.2, here is Thread.sleep
> assertFalse(master.getSnapshotManager().isTakingAnySnapshot());
> {code}
> See
> [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/5227/testReport/junit/org.apache.hadoop.hbase.master.cleaner/TestSnapshotFromMaster/testAsyncSnapshotWillNotBlockSnapshotHFileCleaner/]
>
> [https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.2/lastSuccessfulBuild/artifact/dashboard.html]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23658) Fix flaky TestSnapshotFromMaster
[
https://issues.apache.org/jira/browse/HBASE-23658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011637#comment-17011637
]
Guanghao Zhang commented on HBASE-23658:
Pushed to branch-2.1+. Thanks [~zhangduo] for reviewing.
> Fix flaky TestSnapshotFromMaster
>
>
> Key: HBASE-23658
> URL: https://issues.apache.org/jira/browse/HBASE-23658
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Priority: Major
> Attachments: HBASE-23658.branch-2.2.001.patch
>
>
> testAsyncSnapshotWillNotBlockSnapshotHFileCleaner is flaky. The assert may
> fail.
> {code:java}
> assertTrue(master.getSnapshotManager().isTakingAnySnapshot());
> future.get(); // in branch-2.2, here is Thread.sleep
> assertFalse(master.getSnapshotManager().isTakingAnySnapshot());
> {code}
> See
> [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/5227/testReport/junit/org.apache.hadoop.hbase.master.cleaner/TestSnapshotFromMaster/testAsyncSnapshotWillNotBlockSnapshotHFileCleaner/]
>
> [https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.2/lastSuccessfulBuild/artifact/dashboard.html]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23658) Fix flaky TestSnapshotFromMaster
[
https://issues.apache.org/jira/browse/HBASE-23658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-23658:
---
Fix Version/s: 2.1.9
2.2.3
2.3.0
3.0.0
> Fix flaky TestSnapshotFromMaster
>
>
> Key: HBASE-23658
> URL: https://issues.apache.org/jira/browse/HBASE-23658
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.2.3, 2.1.9
>
> Attachments: HBASE-23658.branch-2.2.001.patch
>
>
> testAsyncSnapshotWillNotBlockSnapshotHFileCleaner is flaky. The assert may
> fail.
> {code:java}
> assertTrue(master.getSnapshotManager().isTakingAnySnapshot());
> future.get(); // in branch-2.2, here is Thread.sleep
> assertFalse(master.getSnapshotManager().isTakingAnySnapshot());
> {code}
> See
> [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/5227/testReport/junit/org.apache.hadoop.hbase.master.cleaner/TestSnapshotFromMaster/testAsyncSnapshotWillNotBlockSnapshotHFileCleaner/]
>
> [https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.2/lastSuccessfulBuild/artifact/dashboard.html]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23651) Region balance throttling can be disabled
[ https://issues.apache.org/jira/browse/HBASE-23651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011324#comment-17011324 ] Guanghao Zhang commented on HBASE-23651: [~binlijin] Please update the Release Note. Thanks. > Region balance throttling can be disabled > - > > Key: HBASE-23651 > URL: https://issues.apache.org/jira/browse/HBASE-23651 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.2.3 >Reporter: Lijin Bin >Assignee: Lijin Bin >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.4 > > > HBASE-17178 Add region balance throttling, but it can not be disabled, > sometimes we need no throttle and balance the cluster as fast as possible. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-23659) BaseLoadBalancer#wouldLowerAvailability should consider region replicas
[ https://issues.apache.org/jira/browse/HBASE-23659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang reassigned HBASE-23659: -- Assignee: Guanghao Zhang > BaseLoadBalancer#wouldLowerAvailability should consider region replicas > --- > > Key: HBASE-23659 > URL: https://issues.apache.org/jira/browse/HBASE-23659 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > > Found this issue when try to fix the flaky unit test TestRegionReplicaSplit. > It may fail as > java.lang.AssertionError: Splitted regions should not be assigned to same > region server. > See > [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/5227/testReport/junit/org.apache.hadoop.hbase.master.assignment/TestRegionReplicaSplit/testRegionReplicaSplitRegionAssignment/]. > > Now wouldLowerAvailability method only consider the primary regions. The > replica region can't assign to same server with primary region. But can be > assigned to same server with other replica regions. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23659) BaseLoadBalancer#wouldLowerAvailability should consider region replicas
[
https://issues.apache.org/jira/browse/HBASE-23659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010594#comment-17010594
]
Guanghao Zhang commented on HBASE-23659:
Found a bug when try to fix this. The primary index may be -1... Then
wouldLowerAvailability will always return true.
{code:java}
int primary = regionIndexToPrimaryIndex[region]; // may be -1
if (contains(primariesOfRegionsPerServer[server], primary)) {
// check for whether there are other servers that we can place this region
for (int i = 0; i < primariesOfRegionsPerServer.length; i++) {
if (i != server && !contains(primariesOfRegionsPerServer[i], primary)) {
LOG.debug("Better server return true, region " + regionInfo + " server "
+ serverName);
return true; // meaning there is a better server
}
}
LOG.debug("No better server return false, region " + regionInfo + " server "
+ serverName);
return false; // there is not a better server to place this
}
{code}
> BaseLoadBalancer#wouldLowerAvailability should consider region replicas
> ---
>
> Key: HBASE-23659
> URL: https://issues.apache.org/jira/browse/HBASE-23659
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Priority: Major
>
> Found this issue when try to fix the flaky unit test TestRegionReplicaSplit.
> It may fail as
> java.lang.AssertionError: Splitted regions should not be assigned to same
> region server.
> See
> [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/5227/testReport/junit/org.apache.hadoop.hbase.master.assignment/TestRegionReplicaSplit/testRegionReplicaSplitRegionAssignment/].
>
> Now wouldLowerAvailability method only consider the primary regions. The
> replica region can't assign to same server with primary region. But can be
> assigned to same server with other replica regions.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23659) BaseLoadBalancer#wouldLowerAvailability should consider region replicas
[ https://issues.apache.org/jira/browse/HBASE-23659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-23659: --- Description: Found this issue when try to fix the flaky unit test TestRegionReplicaSplit. It may fail as java.lang.AssertionError: Splitted regions should not be assigned to same region server. See [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/5227/testReport/junit/org.apache.hadoop.hbase.master.assignment/TestRegionReplicaSplit/testRegionReplicaSplitRegionAssignment/]. Now wouldLowerAvailability method only consider the primary regions. The replica region can't assign to same server with primary region. But can be assigned to same server with other replica regions. was: Found this issue when try to fix the flaky unit test TestRegionReplicaSplit. It may fail as java.lang.AssertionError: Splitted regions should not be assigned to same region server. See [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/5227/testReport/junit/org.apache.hadoop.hbase.master.assignment/TestRegionReplicaSplit/testRegionReplicaSplitRegionAssignment/]. > BaseLoadBalancer#wouldLowerAvailability should consider region replicas > --- > > Key: HBASE-23659 > URL: https://issues.apache.org/jira/browse/HBASE-23659 > Project: HBase > Issue Type: Bug >Reporter: Guanghao Zhang >Priority: Major > > Found this issue when try to fix the flaky unit test TestRegionReplicaSplit. > It may fail as > java.lang.AssertionError: Splitted regions should not be assigned to same > region server. > See > [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/5227/testReport/junit/org.apache.hadoop.hbase.master.assignment/TestRegionReplicaSplit/testRegionReplicaSplitRegionAssignment/]. > > Now wouldLowerAvailability method only consider the primary regions. The > replica region can't assign to same server with primary region. But can be > assigned to same server with other replica regions. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23659) BaseLoadBalancer#wouldLowerAvailability should consider region replicas
Guanghao Zhang created HBASE-23659: -- Summary: BaseLoadBalancer#wouldLowerAvailability should consider region replicas Key: HBASE-23659 URL: https://issues.apache.org/jira/browse/HBASE-23659 Project: HBase Issue Type: Bug Reporter: Guanghao Zhang Found this issue when try to fix the flaky unit test TestRegionReplicaSplit. It may fail as java.lang.AssertionError: Splitted regions should not be assigned to same region server. See [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/5227/testReport/junit/org.apache.hadoop.hbase.master.assignment/TestRegionReplicaSplit/testRegionReplicaSplitRegionAssignment/]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23658) Fix flaky TestSnapshotFromMaster
[
https://issues.apache.org/jira/browse/HBASE-23658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-23658:
---
Attachment: HBASE-23658.branch-2.2.001.patch
> Fix flaky TestSnapshotFromMaster
>
>
> Key: HBASE-23658
> URL: https://issues.apache.org/jira/browse/HBASE-23658
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Priority: Major
> Attachments: HBASE-23658.branch-2.2.001.patch
>
>
> testAsyncSnapshotWillNotBlockSnapshotHFileCleaner is flaky. The assert may
> fail.
> {code:java}
> assertTrue(master.getSnapshotManager().isTakingAnySnapshot());
> future.get(); // in branch-2.2, here is Thread.sleep
> assertFalse(master.getSnapshotManager().isTakingAnySnapshot());
> {code}
> See
> [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/5227/testReport/junit/org.apache.hadoop.hbase.master.cleaner/TestSnapshotFromMaster/testAsyncSnapshotWillNotBlockSnapshotHFileCleaner/]
>
> [https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.2/lastSuccessfulBuild/artifact/dashboard.html]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23658) Fix flaky TestSnapshotFromMaster
[
https://issues.apache.org/jira/browse/HBASE-23658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010229#comment-17010229
]
Guanghao Zhang commented on HBASE-23658:
Use waitfor to make the ut more stable.
> Fix flaky TestSnapshotFromMaster
>
>
> Key: HBASE-23658
> URL: https://issues.apache.org/jira/browse/HBASE-23658
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Priority: Major
>
> testAsyncSnapshotWillNotBlockSnapshotHFileCleaner is flaky. The assert may
> fail.
> {code:java}
> assertTrue(master.getSnapshotManager().isTakingAnySnapshot());
> future.get(); // in branch-2.2, here is Thread.sleep
> assertFalse(master.getSnapshotManager().isTakingAnySnapshot());
> {code}
> See
> [https://builds.apache.org/job/HBase-Flaky-Tests/job/master/5227/testReport/junit/org.apache.hadoop.hbase.master.cleaner/TestSnapshotFromMaster/testAsyncSnapshotWillNotBlockSnapshotHFileCleaner/]
>
> [https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.2/lastSuccessfulBuild/artifact/dashboard.html]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-23658) Fix flaky TestSnapshotFromMaster
Guanghao Zhang created HBASE-23658:
--
Summary: Fix flaky TestSnapshotFromMaster
Key: HBASE-23658
URL: https://issues.apache.org/jira/browse/HBASE-23658
Project: HBase
Issue Type: Bug
Reporter: Guanghao Zhang
testAsyncSnapshotWillNotBlockSnapshotHFileCleaner is flaky. The assert may
fail.
{code:java}
assertTrue(master.getSnapshotManager().isTakingAnySnapshot());
future.get(); // in branch-2.2, here is Thread.sleep
assertFalse(master.getSnapshotManager().isTakingAnySnapshot());
{code}
See
[https://builds.apache.org/job/HBase-Flaky-Tests/job/master/5227/testReport/junit/org.apache.hadoop.hbase.master.cleaner/TestSnapshotFromMaster/testAsyncSnapshotWillNotBlockSnapshotHFileCleaner/]
[https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.2/lastSuccessfulBuild/artifact/dashboard.html]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23655) Fix flaky TestRSGroupsKillRS: should wait the SCP to finish
[ https://issues.apache.org/jira/browse/HBASE-23655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-23655: --- Fix Version/s: 2.2.3 2.3.0 3.0.0 > Fix flaky TestRSGroupsKillRS: should wait the SCP to finish > --- > > Key: HBASE-23655 > URL: https://issues.apache.org/jira/browse/HBASE-23655 > Project: HBase > Issue Type: Bug >Affects Versions: 2.2.2 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.3 > > > In teardownAfterMethod, it will moveTables and delete RSGroups. It will fail > if not wait SCP to finish. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23655) Fix flaky TestRSGroupsKillRS: should wait the SCP to finish
[ https://issues.apache.org/jira/browse/HBASE-23655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010214#comment-17010214 ] Guanghao Zhang commented on HBASE-23655: Ping [~zhangduo] for reviewing. > Fix flaky TestRSGroupsKillRS: should wait the SCP to finish > --- > > Key: HBASE-23655 > URL: https://issues.apache.org/jira/browse/HBASE-23655 > Project: HBase > Issue Type: Bug >Affects Versions: 2.2.2 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.3 > > > In teardownAfterMethod, it will moveTables and delete RSGroups. It will fail > if not wait SCP to finish. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23655) Fix flaky TestRSGroupsKillRS: should wait the SCP to finish
[ https://issues.apache.org/jira/browse/HBASE-23655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010213#comment-17010213 ] Guanghao Zhang commented on HBASE-23655: Test it more than 10 times and passed locally. > Fix flaky TestRSGroupsKillRS: should wait the SCP to finish > --- > > Key: HBASE-23655 > URL: https://issues.apache.org/jira/browse/HBASE-23655 > Project: HBase > Issue Type: Bug >Affects Versions: 2.2.2 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > > In teardownAfterMethod, it will moveTables and delete RSGroups. It will fail > if not wait SCP to finish. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-23655) Fix flaky TestRSGroupsKillRS: should wait the SCP to finish
[ https://issues.apache.org/jira/browse/HBASE-23655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang reassigned HBASE-23655: -- Assignee: Guanghao Zhang > Fix flaky TestRSGroupsKillRS: should wait the SCP to finish > --- > > Key: HBASE-23655 > URL: https://issues.apache.org/jira/browse/HBASE-23655 > Project: HBase > Issue Type: Bug >Affects Versions: 2.2.2 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > > In teardownAfterMethod, it will moveTables and delete RSGroups. It will fail > if not wait SCP to finish. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23655) Fix flaky TestRSGroupsKillRS: should wait the SCP to finish
[ https://issues.apache.org/jira/browse/HBASE-23655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-23655: --- Description: In teardownAfterMethod, it will moveTables and delete RSGroups. It will fail if not wait SCP to finish. > Fix flaky TestRSGroupsKillRS: should wait the SCP to finish > --- > > Key: HBASE-23655 > URL: https://issues.apache.org/jira/browse/HBASE-23655 > Project: HBase > Issue Type: Bug >Affects Versions: 2.2.2 >Reporter: Guanghao Zhang >Priority: Major > > In teardownAfterMethod, it will moveTables and delete RSGroups. It will fail > if not wait SCP to finish. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23655) Fix flaky TestRSGroupsKillRS: should wait the SCP to finish
Guanghao Zhang created HBASE-23655: -- Summary: Fix flaky TestRSGroupsKillRS: should wait the SCP to finish Key: HBASE-23655 URL: https://issues.apache.org/jira/browse/HBASE-23655 Project: HBase Issue Type: Bug Affects Versions: 2.2.2 Reporter: Guanghao Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23638) Set version to 2.2.3 in branch-2.2 for first RC of 2.2.3
[ https://issues.apache.org/jira/browse/HBASE-23638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-23638: --- Fix Version/s: 2.2.3 > Set version to 2.2.3 in branch-2.2 for first RC of 2.2.3 > > > Key: HBASE-23638 > URL: https://issues.apache.org/jira/browse/HBASE-23638 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Priority: Major > Fix For: 2.2.3 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23637) Generate CHANGES.md and RELEASENOTES.md for 2.2.3
[ https://issues.apache.org/jira/browse/HBASE-23637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-23637: --- Fix Version/s: 2.2.3 > Generate CHANGES.md and RELEASENOTES.md for 2.2.3 > - > > Key: HBASE-23637 > URL: https://issues.apache.org/jira/browse/HBASE-23637 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Priority: Major > Fix For: 2.2.3 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23636) Disable table may hang when regionserver stop or abort.
[ https://issues.apache.org/jira/browse/HBASE-23636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17008555#comment-17008555 ] Guanghao Zhang commented on HBASE-23636: Plan to add this fix to 2.2.3. [~binlijin] Please help to commit this. Thanks. > Disable table may hang when regionserver stop or abort. > --- > > Key: HBASE-23636 > URL: https://issues.apache.org/jira/browse/HBASE-23636 > Project: HBase > Issue Type: Bug > Components: amv2 >Affects Versions: 2.2.3 >Reporter: Lijin Bin >Assignee: Lijin Bin >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.3 > > > ServerCrashProcedure will skip regions when table disabling and no procedure > with regions, and when DisableTableProcedure to close this region it will > never success. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23636) Disable table may hang when regionserver stop or abort.
[ https://issues.apache.org/jira/browse/HBASE-23636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-23636: --- Fix Version/s: 2.3.0 3.0.0 > Disable table may hang when regionserver stop or abort. > --- > > Key: HBASE-23636 > URL: https://issues.apache.org/jira/browse/HBASE-23636 > Project: HBase > Issue Type: Bug > Components: amv2 >Affects Versions: 2.2.3 >Reporter: Lijin Bin >Assignee: Lijin Bin >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.3 > > > ServerCrashProcedure will skip regions when table disabling and no procedure > with regions, and when DisableTableProcedure to close this region it will > never success. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23636) Disable table may hang when regionserver stop or abort.
[ https://issues.apache.org/jira/browse/HBASE-23636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-23636: --- Issue Type: Bug (was: Improvement) > Disable table may hang when regionserver stop or abort. > --- > > Key: HBASE-23636 > URL: https://issues.apache.org/jira/browse/HBASE-23636 > Project: HBase > Issue Type: Bug > Components: amv2 >Affects Versions: 2.2.3 >Reporter: Lijin Bin >Assignee: Lijin Bin >Priority: Major > > ServerCrashProcedure will skip regions when table disabling and no procedure > with regions, and when DisableTableProcedure to close this region it will > never success. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23636) Disable table may hang when regionserver stop or abort.
[ https://issues.apache.org/jira/browse/HBASE-23636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-23636: --- Fix Version/s: 2.2.3 > Disable table may hang when regionserver stop or abort. > --- > > Key: HBASE-23636 > URL: https://issues.apache.org/jira/browse/HBASE-23636 > Project: HBase > Issue Type: Bug > Components: amv2 >Affects Versions: 2.2.3 >Reporter: Lijin Bin >Assignee: Lijin Bin >Priority: Major > Fix For: 2.2.3 > > > ServerCrashProcedure will skip regions when table disabling and no procedure > with regions, and when DisableTableProcedure to close this region it will > never success. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23642) Reintroduce ReplicationUtils.contains as deprecated
[ https://issues.apache.org/jira/browse/HBASE-23642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17007849#comment-17007849 ] Guanghao Zhang commented on HBASE-23642: I searched issues yesterday but not found this. Thanks for reminding. :) > Reintroduce ReplicationUtils.contains as deprecated > --- > > Key: HBASE-23642 > URL: https://issues.apache.org/jira/browse/HBASE-23642 > Project: HBase > Issue Type: Task > Components: Replication >Affects Versions: 2.3.0, 2.2.3, 2.1.9 >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Major > Fix For: 2.3.0, 2.2.3, 2.1.9 > > > ReplicationUtils.contains was removed in HBASE-23345. This is an IA.Private > class but HBCK2 relied on this method. > The plan is to add back the method as deprecated and also move HBCK2 off of > this to the new ReplicationPeerConfig#needToReplicate(TableName) instead > (HBASE-23641). > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23642) Reintroduce ReplicationUtils.contains as deprecated
[ https://issues.apache.org/jira/browse/HBASE-23642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17007844#comment-17007844 ] Guanghao Zhang commented on HBASE-23642: OK. +1 for branch-2.2. > Reintroduce ReplicationUtils.contains as deprecated > --- > > Key: HBASE-23642 > URL: https://issues.apache.org/jira/browse/HBASE-23642 > Project: HBase > Issue Type: Task > Components: Replication >Affects Versions: 2.3.0, 2.2.3, 2.1.9 >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Major > Fix For: 2.3.0, 2.2.3, 2.1.9 > > > ReplicationUtils.contains was removed in HBASE-23345. This is an IA.Private > class but HBCK2 relied on this method. > The plan is to add back the method as deprecated and also move HBCK2 off of > this to the new ReplicationPeerConfig#needToReplicate(TableName) instead > (HBASE-23641). > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23638) Set version to 2.2.3 in branch-2.2 for first RC of 2.2.3
Guanghao Zhang created HBASE-23638: -- Summary: Set version to 2.2.3 in branch-2.2 for first RC of 2.2.3 Key: HBASE-23638 URL: https://issues.apache.org/jira/browse/HBASE-23638 Project: HBase Issue Type: Sub-task Reporter: Guanghao Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23637) Generate CHANGES.md and RELEASENOTES.md for 2.2.3
Guanghao Zhang created HBASE-23637: -- Summary: Generate CHANGES.md and RELEASENOTES.md for 2.2.3 Key: HBASE-23637 URL: https://issues.apache.org/jira/browse/HBASE-23637 Project: HBase Issue Type: Sub-task Reporter: Guanghao Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23175) Yarn unable to acquire delegation token for HBase Spark jobs
[
https://issues.apache.org/jira/browse/HBASE-23175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang resolved HBASE-23175.
Resolution: Fixed
Pushed to branch-2.2.
> Yarn unable to acquire delegation token for HBase Spark jobs
>
>
> Key: HBASE-23175
> URL: https://issues.apache.org/jira/browse/HBASE-23175
> Project: HBase
> Issue Type: Bug
> Components: security, spark
>Affects Versions: 2.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.2.3, 2.1.8
>
> Attachments: HBASE-23175.master.001.patch
>
>
> Spark rely on the TokenUtil.obtainToken(conf) API which is removed in
> HBase-2.0, though it has been fixed in SPARK-26432 to use the new API but
> planned for Spark-3.0, hence we need the fix in HBase until they release it
> and we upgrade it
> {code}
> 18/03/20 20:39:07 ERROR ApplicationMaster: User class threw exception:
> org.apache.hadoop.hbase.HBaseIOException:
> com.google.protobuf.ServiceException: Error calling method
> hbase.pb.AuthenticationService.GetAuthenticationToken
> org.apache.hadoop.hbase.HBaseIOException:
> com.google.protobuf.ServiceException: Error calling method
> hbase.pb.AuthenticationService.GetAuthenticationToken
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:360)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:346)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:86)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil$1.run(TokenUtil.java:121)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil$1.run(TokenUtil.java:118)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:313)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:118)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.addTokenForJob(TokenUtil.java:272)
> at
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initCredentials(TableMapReduceUtil.java:533)
> at
> org.apache.hadoop.hbase.spark.HBaseContext.(HBaseContext.scala:73)
> at
> org.apache.hadoop.hbase.spark.JavaHBaseContext.(JavaHBaseContext.scala:46)
> at
> org.apache.hadoop.hbase.spark.example.hbasecontext.JavaHBaseBulkDeleteExample.main(JavaHBaseBulkDeleteExample.java:64)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:706)
> Caused by: com.google.protobuf.ServiceException: Error calling method
> hbase.pb.AuthenticationService.GetAuthenticationToken
> at
> org.apache.hadoop.hbase.client.SyncCoprocessorRpcChannel.callBlockingMethod(SyncCoprocessorRpcChannel.java:71)
> at
> org.apache.hadoop.hbase.protobuf.generated.AuthenticationProtos$AuthenticationService$BlockingStub.getAuthenticationToken(AuthenticationProtos.java:4512)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:81)
> ... 17 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (HBASE-23175) Yarn unable to acquire delegation token for HBase Spark jobs
[
https://issues.apache.org/jira/browse/HBASE-23175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang reopened HBASE-23175:
Forgot to pushed to branch-2.2. Reopened it.
> Yarn unable to acquire delegation token for HBase Spark jobs
>
>
> Key: HBASE-23175
> URL: https://issues.apache.org/jira/browse/HBASE-23175
> Project: HBase
> Issue Type: Bug
> Components: security, spark
>Affects Versions: 2.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3
>
> Attachments: HBASE-23175.master.001.patch
>
>
> Spark rely on the TokenUtil.obtainToken(conf) API which is removed in
> HBase-2.0, though it has been fixed in SPARK-26432 to use the new API but
> planned for Spark-3.0, hence we need the fix in HBase until they release it
> and we upgrade it
> {code}
> 18/03/20 20:39:07 ERROR ApplicationMaster: User class threw exception:
> org.apache.hadoop.hbase.HBaseIOException:
> com.google.protobuf.ServiceException: Error calling method
> hbase.pb.AuthenticationService.GetAuthenticationToken
> org.apache.hadoop.hbase.HBaseIOException:
> com.google.protobuf.ServiceException: Error calling method
> hbase.pb.AuthenticationService.GetAuthenticationToken
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:360)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:346)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:86)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil$1.run(TokenUtil.java:121)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil$1.run(TokenUtil.java:118)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:313)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:118)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.addTokenForJob(TokenUtil.java:272)
> at
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initCredentials(TableMapReduceUtil.java:533)
> at
> org.apache.hadoop.hbase.spark.HBaseContext.(HBaseContext.scala:73)
> at
> org.apache.hadoop.hbase.spark.JavaHBaseContext.(JavaHBaseContext.scala:46)
> at
> org.apache.hadoop.hbase.spark.example.hbasecontext.JavaHBaseBulkDeleteExample.main(JavaHBaseBulkDeleteExample.java:64)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:706)
> Caused by: com.google.protobuf.ServiceException: Error calling method
> hbase.pb.AuthenticationService.GetAuthenticationToken
> at
> org.apache.hadoop.hbase.client.SyncCoprocessorRpcChannel.callBlockingMethod(SyncCoprocessorRpcChannel.java:71)
> at
> org.apache.hadoop.hbase.protobuf.generated.AuthenticationProtos$AuthenticationService$BlockingStub.getAuthenticationToken(AuthenticationProtos.java:4512)
> at
> org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:81)
> ... 17 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-23553) Snapshot referenced data files are deleted in some case
[
https://issues.apache.org/jira/browse/HBASE-23553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang resolved HBASE-23553.
Fix Version/s: 2.2.3
2.3.0
3.0.0
Resolution: Fixed
> Snapshot referenced data files are deleted in some case
> ---
>
> Key: HBASE-23553
> URL: https://issues.apache.org/jira/browse/HBASE-23553
> Project: HBase
> Issue Type: Bug
>Reporter: Yi Mei
>Assignee: Yi Mei
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.2.3
>
>
> We scan snapshot in our cluster and got following exception:
> {code:java}
> java.io.IOException: java.io.IOException: java.io.FileNotFoundException:
> Unable to open link: org.apache.hadoop.hbase.io.HFileLink
> locations=[hdfs://tjwqsrv-galaxy98/hbase/tjwqsrv-galaxy98/data/default/galaxy_online_fds_object_table/06dd90d8540b56343859b63a6134450c/A/4a6cf05f419a9f61059cb05a962f,
>
> hdfs://tjwqsrv-galaxy98/hbase/tjwqsrv-galaxy98/.tmp/data/default/galaxy_online_fds_object_table/06dd90d8540b56343859b63a6134450c/A/4a6cf05f419a9f61059cb05a962f,
>
> hdfs://tjwqsrv-galaxy98/hbase/tjwqsrv-galaxy98/archive/data/default/galaxy_online_fds_object_table/06dd90d8540b56343859b63a6134450c/A/4a6cf05f419a9f61059cb05a962f]
>
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionStores(HRegion.java:867)
>
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:778)
> at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:749)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5306)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5271)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5243)
> at
> org.apache.hadoop.hbase.client.ClientSideRegionScanner.(ClientSideRegionScanner.java:72)
>
> at
> org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormatImpl$RecordReader.initialize(TableSnapshotInputFormatImpl.java:239)
>
> at
> org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormat$TableSnapshotRegionRecordReader.initialize(TableSnapshotInputFormat.java:150)
>
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:552)
> at {code}
> I checked to namenode logs and found that this file is deleted by hbase
> cleaner although a snapshot still referenced to this file.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-23231) ReplicationSource do not update metrics after refresh
[
https://issues.apache.org/jira/browse/HBASE-23231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-23231:
---
Fix Version/s: (was: 2.2.2)
2.2.3
> ReplicationSource do not update metrics after refresh
> -
>
> Key: HBASE-23231
> URL: https://issues.apache.org/jira/browse/HBASE-23231
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.2.2
>Reporter: Lijin Bin
>Assignee: Lijin Bin
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3
>
>
> When replication refresh to new state, it will create a new source and
> terminate the old source and replace the old source with new source.
> {code}
> public void refreshSources(String peerId) throws IOException {
> String terminateMessage = "Peer " + peerId +
> " state or config changed. Will close the previous replication source
> and open a new one";
> ReplicationPeer peer = replicationPeers.getPeer(peerId);
> ReplicationSourceInterface src = createSource(peerId, peer);
> // synchronized on latestPaths to avoid missing the new log
> synchronized (this.latestPaths) {
> ReplicationSourceInterface toRemove = this.sources.put(peerId, src);
> if (toRemove != null) {
> LOG.info("Terminate replication source for " + toRemove.getPeerId());
> toRemove.terminate(terminateMessage);
> }
> for (NavigableSet walsByGroup : walsById.get(peerId).values()) {
> walsByGroup.forEach(wal -> src.enqueueLog(new Path(this.logDir,
> wal)));
> }
> }
> LOG.info("Startup replication source for " + src.getPeerId());
> src.startup();
> {code}
> terminate replication source will remove all metrics, current terminate
> replication source be called after create new source which do init metrics,
> so the result is there is no corresponding metrics after refresh replication
> source.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-23286) Improve MTTR: Split WAL to HFile
[ https://issues.apache.org/jira/browse/HBASE-23286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang resolved HBASE-23286. Resolution: Fixed Pushed to branch-2 and master. Thanks all for reviewing. And opened two follow-up issues. > Improve MTTR: Split WAL to HFile > > > Key: HBASE-23286 > URL: https://issues.apache.org/jira/browse/HBASE-23286 > Project: HBase > Issue Type: Improvement > Components: MTTR >Affects Versions: 3.0.0, 2.3.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > After HBASE-20724, the compaction event marker is not used anymore when > failover. So our new proposal is split WAL to HFile to imporve MTTR. It has 3 > steps: > # Read WAL and write HFile to region’s column family’s recovered.hfiles > directory. > # Open region. > # Bulkload the recovered.hfiles for every column family. > The design doc was attathed by a google doc. Any suggestions are welcomed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23634) Enable "Split WAL to HFile" by default
Guanghao Zhang created HBASE-23634: -- Summary: Enable "Split WAL to HFile" by default Key: HBASE-23634 URL: https://issues.apache.org/jira/browse/HBASE-23634 Project: HBase Issue Type: Improvement Affects Versions: 3.0.0, 2.3.0 Reporter: Guanghao Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23633) Find a way to handle the corrupt recovered hfiles
Guanghao Zhang created HBASE-23633: -- Summary: Find a way to handle the corrupt recovered hfiles Key: HBASE-23633 URL: https://issues.apache.org/jira/browse/HBASE-23633 Project: HBase Issue Type: Umbrella Reporter: Guanghao Zhang Copy the comment from PR review. If the file is a corrupt HFile, an exception will be thrown here, which will cause the region to fail to open. Maybe we can add a new parameter to control whether to skip the exception, similar to recover edits which has a parameter "hbase.hregion.edits.replay.skip.errors"; Regions that can't be opened because of detached References or corrupt hfiles are a fact-of-life. We need work on this issue. This will be a new variant on the problem -- i.e. bad recovered hfiles. On adding a config to ignore bad files and just open, thats a bit dangerous as per @infraio as it could mean silent data loss. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23633) Find a way to handle the corrupt recovered hfiles
[ https://issues.apache.org/jira/browse/HBASE-23633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-23633: --- Issue Type: Improvement (was: Umbrella) > Find a way to handle the corrupt recovered hfiles > - > > Key: HBASE-23633 > URL: https://issues.apache.org/jira/browse/HBASE-23633 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Priority: Major > > Copy the comment from PR review. > > If the file is a corrupt HFile, an exception will be thrown here, which will > cause the region to fail to open. > Maybe we can add a new parameter to control whether to skip the exception, > similar to recover edits which has a parameter > "hbase.hregion.edits.replay.skip.errors"; > > Regions that can't be opened because of detached References or corrupt hfiles > are a fact-of-life. We need work on this issue. This will be a new variant on > the problem -- i.e. bad recovered hfiles. > On adding a config to ignore bad files and just open, thats a bit dangerous > as per @infraio as it could mean silent data loss. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23286) Improve MTTR: Split WAL to HFile
[ https://issues.apache.org/jira/browse/HBASE-23286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-23286: --- Release Note: Add a new feature to improve MTTR which have 3 steps to failover: 1. Read WAL and write HFile to region’s column family’s recovered.hfiles directory. 2. Open region. 3. Bulkload the recovered.hfiles for every column family. Compared to DLS(distributed log split), this feature will reduce region open time significantly. Config hbase.wal.split.to.hfile to true to enable this featue. > Improve MTTR: Split WAL to HFile > > > Key: HBASE-23286 > URL: https://issues.apache.org/jira/browse/HBASE-23286 > Project: HBase > Issue Type: Improvement > Components: MTTR >Affects Versions: 3.0.0, 2.3.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0, 2.3.0 > > > After HBASE-20724, the compaction event marker is not used anymore when > failover. So our new proposal is split WAL to HFile to imporve MTTR. It has 3 > steps: > # Read WAL and write HFile to region’s column family’s recovered.hfiles > directory. > # Open region. > # Bulkload the recovered.hfiles for every column family. > The design doc was attathed by a google doc. Any suggestions are welcomed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
