Re: [DISCUSS] Time to release 2.5.3 and 2.4.16?

[Tak Lon (Stephen) Wu]

Thanks Andrew, I merged it and currently have been running the
do-release-docker.sh (with -f publish) for 2+ hours, will check again
tomorrow.

-Stephen

On Tue, Jan 24, 2023 at 5:24 PM Andrew Purtell  wrote:
>
> Approved the PR
>
> On Tue, Jan 24, 2023 at 2:53 PM Tak Lon (Stephen) Wu 
> wrote:
>
> > nvmI found the issue and it's just a duplicated test classname
> > causing the NullPointerException.
> >
> > filed https://issues.apache.org/jira/browse/HBASE-27589, please help
> > to review it.
> >
> > -Stephen
> >
> > On Tue, Jan 24, 2023 at 2:17 PM Tak Lon (Stephen) Wu 
> > wrote:
> > >
> > > During the execution of publish-dist (part of the do-release.sh), I
> > > found the strange problem with commit
> > > 748ec455b33d29ed007b0143efa73b7409248da7 from HBASE-27498, the test in
> > >
> > hbase-it/src/test/java/org/apache/hadoop/hbase/client/TestConnectionImplementation.java
> > > is blocking the javadoc target test-aggregate-no-fork when running mvn
> > > -B --settings  site -DskipTests with the
> > > following ERROR.
> > >
> > > I tried to revert 748ec455b33d29ed007b0143efa73b7409248da7 or revert
> > > just the TestConnectionImplementation class, then the javadoc/site
> > > completes without error. I also tried to checkout just branch-2.5 and
> > > branch-2.4, execute the build and javadoc/site execution, but it is
> > > still failing.
> > >
> > > # capture using JDK17 , it has the same NullPointerException exception
> > > for JDK8 and JDK11 without much detail.
> > > error: An internal exception has occurred.
> > >   (java.lang.NullPointerException: Cannot invoke
> > > "com.sun.source.util.TreePath.getCompilationUnit()" because the return
> > > value of
> > "com.sun.source.util.DocTrees.getPath(javax.lang.model.element.Element)"
> > > is null)
> > > Please file a bug against the javadoc tool via the Java bug reporting
> > page
> > > (https://bugreport.java.com) after checking the Bug Database
> > > (https://bugs.java.com)
> > > for duplicates. Include error messages and the following diagnostic in
> > > your report. Thank you.
> > > java.lang.NullPointerException: Cannot invoke
> > > "com.sun.source.util.TreePath.getCompilationUnit()" because the return
> > > value of
> > "com.sun.source.util.DocTrees.getPath(javax.lang.model.element.Element)"
> > > is null
> > > at
> > jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.util.Utils.getFileObject(Utils.java:871)
> > > at
> > jdk.javadoc/jdk.javadoc.internal.doclets.formats.html.SourceToHTMLConverter.convertClass(SourceToHTMLConverter.java:198)
> > > at
> > jdk.javadoc/jdk.javadoc.internal.doclets.formats.html.SourceToHTMLConverter.generate(SourceToHTMLConverter.java:137)
> > > at
> > jdk.javadoc/jdk.javadoc.internal.doclets.formats.html.SourceToHTMLConverter.convertRoot(SourceToHTMLConverter.java:112)
> > > at
> > jdk.javadoc/jdk.javadoc.internal.doclets.formats.html.HtmlDoclet.generateOtherFiles(HtmlDoclet.java:218)
> > > at
> > jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.AbstractDoclet.startGeneration(AbstractDoclet.java:218)
> > > at
> > jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.AbstractDoclet.run(AbstractDoclet.java:115)
> > > at
> > jdk.javadoc/jdk.javadoc.doclet.StandardDoclet.run(StandardDoclet.java:103)
> > > at
> > jdk.javadoc/jdk.javadoc.internal.tool.Start.parseAndExecute(Start.java:556)
> > > at jdk.javadoc/jdk.javadoc.internal.tool.Start.begin(Start.java:393)
> > > at jdk.javadoc/jdk.javadoc.internal.tool.Start.begin(Start.java:342)
> > > at jdk.javadoc/jdk.javadoc.internal.tool.Main.execute(Main.java:63)
> > > at jdk.javadoc/jdk.javadoc.internal.tool.Main.main(Main.java:52)
> > > 1 error
> > > 100 warnings
> > >
> > > May I ask if you checkout the branch-2.5 and run the following to see
> > > if I'm the only one hitting the error ? the command is mvn clean
> > > install -DskipTests && mvn -B site -DskipTests
> > >
> > > Secondly, other than reverting this change, do you have any suggestion
> > > how we can debug javadoc generation (I tried javadoc -J-Xmx2G @options
> > > @argfile -verbose, but this didn't tell me anything...)
> > >
> > > -Stephen
> > >
> > > On Mon, Jan 23, 2023 at 9:01 PM Tak Lon (Stephen) Wu 
> > wrote:
> > > >
> > > > Fyi I’m executing the release build via do-release-docker.sh, ran till
> > the mvn site and Java doc generation.
> > > >
> > > > But it failed with 20:05:46 [ERROR] javadoc: error -
> > java.lang.NullPointerException , and few error/warning for @result tag as
> > well as @link not found for the generated proto class BlockingInterface.
> > > >
> > > > I’m still working on it, if possible, please avoid to push commits to
> > branch-2.5
> > > >
> > > > Thanks,
> > > > Stephen
> > > >
> > > > On Sun, Jan 22, 2023 at 1:29 PM Andrew Purtell 
> > wrote:
> > > >>
> > > >> Sounds good Stephen, please ping me as you need assistance.
> > > >>
> > > >> On Sat, Jan 21, 2023 at 9:32 PM Tak Lon (Stephen) Wu <
> > tak...@apache.org>
> > > >> wrote:
> > > >>
> > > >> > Thanks Andrew and Bryan, all the blockers 

Re: [DISCUSS] Time to release 2.5.3 and 2.4.16?

[Andrew Purtell]

Approved the PR

On Tue, Jan 24, 2023 at 2:53 PM Tak Lon (Stephen) Wu 
wrote:

> nvmI found the issue and it's just a duplicated test classname
> causing the NullPointerException.
>
> filed https://issues.apache.org/jira/browse/HBASE-27589, please help
> to review it.
>
> -Stephen
>
> On Tue, Jan 24, 2023 at 2:17 PM Tak Lon (Stephen) Wu 
> wrote:
> >
> > During the execution of publish-dist (part of the do-release.sh), I
> > found the strange problem with commit
> > 748ec455b33d29ed007b0143efa73b7409248da7 from HBASE-27498, the test in
> >
> hbase-it/src/test/java/org/apache/hadoop/hbase/client/TestConnectionImplementation.java
> > is blocking the javadoc target test-aggregate-no-fork when running mvn
> > -B --settings  site -DskipTests with the
> > following ERROR.
> >
> > I tried to revert 748ec455b33d29ed007b0143efa73b7409248da7 or revert
> > just the TestConnectionImplementation class, then the javadoc/site
> > completes without error. I also tried to checkout just branch-2.5 and
> > branch-2.4, execute the build and javadoc/site execution, but it is
> > still failing.
> >
> > # capture using JDK17 , it has the same NullPointerException exception
> > for JDK8 and JDK11 without much detail.
> > error: An internal exception has occurred.
> >   (java.lang.NullPointerException: Cannot invoke
> > "com.sun.source.util.TreePath.getCompilationUnit()" because the return
> > value of
> "com.sun.source.util.DocTrees.getPath(javax.lang.model.element.Element)"
> > is null)
> > Please file a bug against the javadoc tool via the Java bug reporting
> page
> > (https://bugreport.java.com) after checking the Bug Database
> > (https://bugs.java.com)
> > for duplicates. Include error messages and the following diagnostic in
> > your report. Thank you.
> > java.lang.NullPointerException: Cannot invoke
> > "com.sun.source.util.TreePath.getCompilationUnit()" because the return
> > value of
> "com.sun.source.util.DocTrees.getPath(javax.lang.model.element.Element)"
> > is null
> > at
> jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.util.Utils.getFileObject(Utils.java:871)
> > at
> jdk.javadoc/jdk.javadoc.internal.doclets.formats.html.SourceToHTMLConverter.convertClass(SourceToHTMLConverter.java:198)
> > at
> jdk.javadoc/jdk.javadoc.internal.doclets.formats.html.SourceToHTMLConverter.generate(SourceToHTMLConverter.java:137)
> > at
> jdk.javadoc/jdk.javadoc.internal.doclets.formats.html.SourceToHTMLConverter.convertRoot(SourceToHTMLConverter.java:112)
> > at
> jdk.javadoc/jdk.javadoc.internal.doclets.formats.html.HtmlDoclet.generateOtherFiles(HtmlDoclet.java:218)
> > at
> jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.AbstractDoclet.startGeneration(AbstractDoclet.java:218)
> > at
> jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.AbstractDoclet.run(AbstractDoclet.java:115)
> > at
> jdk.javadoc/jdk.javadoc.doclet.StandardDoclet.run(StandardDoclet.java:103)
> > at
> jdk.javadoc/jdk.javadoc.internal.tool.Start.parseAndExecute(Start.java:556)
> > at jdk.javadoc/jdk.javadoc.internal.tool.Start.begin(Start.java:393)
> > at jdk.javadoc/jdk.javadoc.internal.tool.Start.begin(Start.java:342)
> > at jdk.javadoc/jdk.javadoc.internal.tool.Main.execute(Main.java:63)
> > at jdk.javadoc/jdk.javadoc.internal.tool.Main.main(Main.java:52)
> > 1 error
> > 100 warnings
> >
> > May I ask if you checkout the branch-2.5 and run the following to see
> > if I'm the only one hitting the error ? the command is mvn clean
> > install -DskipTests && mvn -B site -DskipTests
> >
> > Secondly, other than reverting this change, do you have any suggestion
> > how we can debug javadoc generation (I tried javadoc -J-Xmx2G @options
> > @argfile -verbose, but this didn't tell me anything...)
> >
> > -Stephen
> >
> > On Mon, Jan 23, 2023 at 9:01 PM Tak Lon (Stephen) Wu 
> wrote:
> > >
> > > Fyi I’m executing the release build via do-release-docker.sh, ran till
> the mvn site and Java doc generation.
> > >
> > > But it failed with 20:05:46 [ERROR] javadoc: error -
> java.lang.NullPointerException , and few error/warning for @result tag as
> well as @link not found for the generated proto class BlockingInterface.
> > >
> > > I’m still working on it, if possible, please avoid to push commits to
> branch-2.5
> > >
> > > Thanks,
> > > Stephen
> > >
> > > On Sun, Jan 22, 2023 at 1:29 PM Andrew Purtell 
> wrote:
> > >>
> > >> Sounds good Stephen, please ping me as you need assistance.
> > >>
> > >> On Sat, Jan 21, 2023 at 9:32 PM Tak Lon (Stephen) Wu <
> tak...@apache.org>
> > >> wrote:
> > >>
> > >> > Thanks Andrew and Bryan, all the blockers for branch-2.5 should have
> > >> > been resolved.
> > >> >
> > >> > I'm running the compatibility check again, and will continue the
> > >> > release tasks tomorrow or Monday morning (PST).
> > >> >
> > >> > -Stephen
> > >> >
> > >> > On Fri, Jan 20, 2023 at 11:16 AM Bryan Beaudreault
> > >> >  wrote:
> > >> > >
> > >> > > You probably saw, but just to close the loop here -- HBASE-27579
> got
> > >> > > resolved, so no blocker 

Re: [DISCUSS] Time to release 2.5.3 and 2.4.16?

[Tak Lon (Stephen) Wu]

nvmI found the issue and it's just a duplicated test classname
causing the NullPointerException.

filed https://issues.apache.org/jira/browse/HBASE-27589, please help
to review it.

-Stephen

On Tue, Jan 24, 2023 at 2:17 PM Tak Lon (Stephen) Wu  wrote:
>
> During the execution of publish-dist (part of the do-release.sh), I
> found the strange problem with commit
> 748ec455b33d29ed007b0143efa73b7409248da7 from HBASE-27498, the test in
> hbase-it/src/test/java/org/apache/hadoop/hbase/client/TestConnectionImplementation.java
> is blocking the javadoc target test-aggregate-no-fork when running mvn
> -B --settings  site -DskipTests with the
> following ERROR.
>
> I tried to revert 748ec455b33d29ed007b0143efa73b7409248da7 or revert
> just the TestConnectionImplementation class, then the javadoc/site
> completes without error. I also tried to checkout just branch-2.5 and
> branch-2.4, execute the build and javadoc/site execution, but it is
> still failing.
>
> # capture using JDK17 , it has the same NullPointerException exception
> for JDK8 and JDK11 without much detail.
> error: An internal exception has occurred.
>   (java.lang.NullPointerException: Cannot invoke
> "com.sun.source.util.TreePath.getCompilationUnit()" because the return
> value of 
> "com.sun.source.util.DocTrees.getPath(javax.lang.model.element.Element)"
> is null)
> Please file a bug against the javadoc tool via the Java bug reporting page
> (https://bugreport.java.com) after checking the Bug Database
> (https://bugs.java.com)
> for duplicates. Include error messages and the following diagnostic in
> your report. Thank you.
> java.lang.NullPointerException: Cannot invoke
> "com.sun.source.util.TreePath.getCompilationUnit()" because the return
> value of 
> "com.sun.source.util.DocTrees.getPath(javax.lang.model.element.Element)"
> is null
> at 
> jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.util.Utils.getFileObject(Utils.java:871)
> at 
> jdk.javadoc/jdk.javadoc.internal.doclets.formats.html.SourceToHTMLConverter.convertClass(SourceToHTMLConverter.java:198)
> at 
> jdk.javadoc/jdk.javadoc.internal.doclets.formats.html.SourceToHTMLConverter.generate(SourceToHTMLConverter.java:137)
> at 
> jdk.javadoc/jdk.javadoc.internal.doclets.formats.html.SourceToHTMLConverter.convertRoot(SourceToHTMLConverter.java:112)
> at 
> jdk.javadoc/jdk.javadoc.internal.doclets.formats.html.HtmlDoclet.generateOtherFiles(HtmlDoclet.java:218)
> at 
> jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.AbstractDoclet.startGeneration(AbstractDoclet.java:218)
> at 
> jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.AbstractDoclet.run(AbstractDoclet.java:115)
> at jdk.javadoc/jdk.javadoc.doclet.StandardDoclet.run(StandardDoclet.java:103)
> at jdk.javadoc/jdk.javadoc.internal.tool.Start.parseAndExecute(Start.java:556)
> at jdk.javadoc/jdk.javadoc.internal.tool.Start.begin(Start.java:393)
> at jdk.javadoc/jdk.javadoc.internal.tool.Start.begin(Start.java:342)
> at jdk.javadoc/jdk.javadoc.internal.tool.Main.execute(Main.java:63)
> at jdk.javadoc/jdk.javadoc.internal.tool.Main.main(Main.java:52)
> 1 error
> 100 warnings
>
> May I ask if you checkout the branch-2.5 and run the following to see
> if I'm the only one hitting the error ? the command is mvn clean
> install -DskipTests && mvn -B site -DskipTests
>
> Secondly, other than reverting this change, do you have any suggestion
> how we can debug javadoc generation (I tried javadoc -J-Xmx2G @options
> @argfile -verbose, but this didn't tell me anything...)
>
> -Stephen
>
> On Mon, Jan 23, 2023 at 9:01 PM Tak Lon (Stephen) Wu  
> wrote:
> >
> > Fyi I’m executing the release build via do-release-docker.sh, ran till the 
> > mvn site and Java doc generation.
> >
> > But it failed with 20:05:46 [ERROR] javadoc: error - 
> > java.lang.NullPointerException , and few error/warning for @result tag as 
> > well as @link not found for the generated proto class BlockingInterface.
> >
> > I’m still working on it, if possible, please avoid to push commits to 
> > branch-2.5
> >
> > Thanks,
> > Stephen
> >
> > On Sun, Jan 22, 2023 at 1:29 PM Andrew Purtell  wrote:
> >>
> >> Sounds good Stephen, please ping me as you need assistance.
> >>
> >> On Sat, Jan 21, 2023 at 9:32 PM Tak Lon (Stephen) Wu 
> >> wrote:
> >>
> >> > Thanks Andrew and Bryan, all the blockers for branch-2.5 should have
> >> > been resolved.
> >> >
> >> > I'm running the compatibility check again, and will continue the
> >> > release tasks tomorrow or Monday morning (PST).
> >> >
> >> > -Stephen
> >> >
> >> > On Fri, Jan 20, 2023 at 11:16 AM Bryan Beaudreault
> >> >  wrote:
> >> > >
> >> > > You probably saw, but just to close the loop here -- HBASE-27579 got
> >> > > resolved, so no blocker there anymore. Thanks all!
> >> > >
> >> > > On Fri, Jan 20, 2023 at 12:30 PM Andrew Purtell 
> >> > wrote:
> >> > >
> >> > > > I think HBASE-27539 should be reverted from the releasing branches.
> >> > Keeping
> >> > > > it in branch-2 and master would be fine, we do frequently 

[jira] [Created] (HBASE-27589) Rename TestConnectionImplementationCacheMasterState to fix javadoc failure

[Tak-Lon (Stephen) Wu (Jira)]

Tak-Lon (Stephen) Wu created HBASE-27589:


 Summary: Rename TestConnectionImplementationCacheMasterState to 
fix javadoc failure
 Key: HBASE-27589
 URL: https://issues.apache.org/jira/browse/HBASE-27589
 Project: HBase
  Issue Type: Bug
  Components: Client, documentation
Affects Versions: 2.6.0, 2.4.16, 2.5.3
Reporter: Tak-Lon (Stephen) Wu
Assignee: Tak-Lon (Stephen) Wu


discussed during release of 2.5.3RC0 
[https://lists.apache.org/thread/b34lgz3yy8vkv4fbbxj4mvtjyjrp4m6p]

I found a javadoc problem introduced by HBASE-27498 with the same test class 
shared the same package prefix such that the javadoc failed with a unclear 
java.lang.NullPointerException. 

- 
hbase-it/src/test/java/org/apache/hadoop/hbase/client/TestConnectionImplementation.java
 
- 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestConnectionImplementation.java

after renaming 
hbase-it/src/test/java/org/apache/hadoop/hbase/client/TestConnectionImplementation.java
 to 
hbase-it/src/test/java/org/apache/hadoop/hbase/client/TestConnectionImplementationCacheMasterState.java
 should fix this issue.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Time to release 2.5.3 and 2.4.16?

[Tak Lon (Stephen) Wu]

During the execution of publish-dist (part of the do-release.sh), I
found the strange problem with commit
748ec455b33d29ed007b0143efa73b7409248da7 from HBASE-27498, the test in
hbase-it/src/test/java/org/apache/hadoop/hbase/client/TestConnectionImplementation.java
is blocking the javadoc target test-aggregate-no-fork when running mvn
-B --settings  site -DskipTests with the
following ERROR.

I tried to revert 748ec455b33d29ed007b0143efa73b7409248da7 or revert
just the TestConnectionImplementation class, then the javadoc/site
completes without error. I also tried to checkout just branch-2.5 and
branch-2.4, execute the build and javadoc/site execution, but it is
still failing.

# capture using JDK17 , it has the same NullPointerException exception
for JDK8 and JDK11 without much detail.
error: An internal exception has occurred.
  (java.lang.NullPointerException: Cannot invoke
"com.sun.source.util.TreePath.getCompilationUnit()" because the return
value of 
"com.sun.source.util.DocTrees.getPath(javax.lang.model.element.Element)"
is null)
Please file a bug against the javadoc tool via the Java bug reporting page
(https://bugreport.java.com) after checking the Bug Database
(https://bugs.java.com)
for duplicates. Include error messages and the following diagnostic in
your report. Thank you.
java.lang.NullPointerException: Cannot invoke
"com.sun.source.util.TreePath.getCompilationUnit()" because the return
value of 
"com.sun.source.util.DocTrees.getPath(javax.lang.model.element.Element)"
is null
at 
jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.util.Utils.getFileObject(Utils.java:871)
at 
jdk.javadoc/jdk.javadoc.internal.doclets.formats.html.SourceToHTMLConverter.convertClass(SourceToHTMLConverter.java:198)
at 
jdk.javadoc/jdk.javadoc.internal.doclets.formats.html.SourceToHTMLConverter.generate(SourceToHTMLConverter.java:137)
at 
jdk.javadoc/jdk.javadoc.internal.doclets.formats.html.SourceToHTMLConverter.convertRoot(SourceToHTMLConverter.java:112)
at 
jdk.javadoc/jdk.javadoc.internal.doclets.formats.html.HtmlDoclet.generateOtherFiles(HtmlDoclet.java:218)
at 
jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.AbstractDoclet.startGeneration(AbstractDoclet.java:218)
at 
jdk.javadoc/jdk.javadoc.internal.doclets.toolkit.AbstractDoclet.run(AbstractDoclet.java:115)
at jdk.javadoc/jdk.javadoc.doclet.StandardDoclet.run(StandardDoclet.java:103)
at jdk.javadoc/jdk.javadoc.internal.tool.Start.parseAndExecute(Start.java:556)
at jdk.javadoc/jdk.javadoc.internal.tool.Start.begin(Start.java:393)
at jdk.javadoc/jdk.javadoc.internal.tool.Start.begin(Start.java:342)
at jdk.javadoc/jdk.javadoc.internal.tool.Main.execute(Main.java:63)
at jdk.javadoc/jdk.javadoc.internal.tool.Main.main(Main.java:52)
1 error
100 warnings

May I ask if you checkout the branch-2.5 and run the following to see
if I'm the only one hitting the error ? the command is mvn clean
install -DskipTests && mvn -B site -DskipTests

Secondly, other than reverting this change, do you have any suggestion
how we can debug javadoc generation (I tried javadoc -J-Xmx2G @options
@argfile -verbose, but this didn't tell me anything...)

-Stephen

On Mon, Jan 23, 2023 at 9:01 PM Tak Lon (Stephen) Wu  wrote:
>
> Fyi I’m executing the release build via do-release-docker.sh, ran till the 
> mvn site and Java doc generation.
>
> But it failed with 20:05:46 [ERROR] javadoc: error - 
> java.lang.NullPointerException , and few error/warning for @result tag as 
> well as @link not found for the generated proto class BlockingInterface.
>
> I’m still working on it, if possible, please avoid to push commits to 
> branch-2.5
>
> Thanks,
> Stephen
>
> On Sun, Jan 22, 2023 at 1:29 PM Andrew Purtell  wrote:
>>
>> Sounds good Stephen, please ping me as you need assistance.
>>
>> On Sat, Jan 21, 2023 at 9:32 PM Tak Lon (Stephen) Wu 
>> wrote:
>>
>> > Thanks Andrew and Bryan, all the blockers for branch-2.5 should have
>> > been resolved.
>> >
>> > I'm running the compatibility check again, and will continue the
>> > release tasks tomorrow or Monday morning (PST).
>> >
>> > -Stephen
>> >
>> > On Fri, Jan 20, 2023 at 11:16 AM Bryan Beaudreault
>> >  wrote:
>> > >
>> > > You probably saw, but just to close the loop here -- HBASE-27579 got
>> > > resolved, so no blocker there anymore. Thanks all!
>> > >
>> > > On Fri, Jan 20, 2023 at 12:30 PM Andrew Purtell 
>> > wrote:
>> > >
>> > > > I think HBASE-27539 should be reverted from the releasing branches.
>> > Keeping
>> > > > it in branch-2 and master would be fine, we do frequently change LP
>> > > > interface details in minor releases. It is the compatibility break in a
>> > > > patch release with a real affected downstream (Phoenix) that is a
>> > problem.
>> > > > As RM I would go ahead and do that, I suggest the same to you.
>> > > >
>> > > > On Thu, Jan 19, 2023 at 10:25 PM Tak Lon (Stephen) Wu <
>> > tak...@apache.org>
>> > > > wrote:
>> > > >
>> > > > > Thanks Andrew for sharing the normal flow and helped me off this
>> > > > > thread. I 

[jira] [Created] (HBASE-27588) "Iinstantiating StoreFileTracker impl" INFO level logging is too chatty

[Andrew Kyle Purtell (Jira)]

Andrew Kyle Purtell created HBASE-27588:
---

 Summary: "Iinstantiating StoreFileTracker impl" INFO level logging 
is too chatty
 Key: HBASE-27588
 URL: https://issues.apache.org/jira/browse/HBASE-27588
 Project: HBase
  Issue Type: Task
Reporter: Andrew Kyle Purtell
Assignee: Andrew Kyle Purtell
 Fix For: 2.4.16, 2.5.4


2023-01-24 14:05:17,600 INFO  [PEWorker-9] 
storefiletracker.StoreFileTrackerFactory: instantiating StoreFileTracker impl  
 

This should be logged at DEBUG level as it is debug information. It is 
frequently printed in the logs and because it's the same impl every time 
(although in theory could be varied in table schema) there is no value for 
operators, it just consumes logging resources.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HBASE-27587) L1 cache leaks index blocks over time when under subscribed

[Bryan Beaudreault (Jira)]

Bryan Beaudreault created HBASE-27587:
-

 Summary: L1 cache leaks index blocks over time when under 
subscribed
 Key: HBASE-27587
 URL: https://issues.apache.org/jira/browse/HBASE-27587
 Project: HBase
  Issue Type: Bug
Reporter: Bryan Beaudreault


Let's say you have CombinedBlockCache enabled. DATA goes to L2, INDEX/BLOOM go 
to L1.  Your regionserver has index size of 2gb and bloom size of 1gb. So you 
really only need around 3gb of L1 to fully hold all of the "L1 candidates".

When data set does not fit into cache, LRU will handle evictions to stay under 
max. But in the above scenario, if you configure 6gb for L1 (3 more than 
needed) over time you will end up filling that entire 6gb with old INDEX 
blocks. Once you reach max, LRU will handle evicting out the oldest ones.

Since the leak is contained to the configured max L1 size, this isn't a huge 
issue but it results in heap waste. Under high heap allocations, if you haven't 
left enough buffer outside memstore, L1, etc, you will start seeing GC 
pressure. This L1 leak then becomes a little more problematic, because you end 
up in a circumstance where longer lived regionservers (who've leaked closer to 
the max L1 size) have less extra buffer available than more newly restarted 
regionservers.

The best fix is to appropriately set your L1 size so there is not a lot of 
excess, but this can be painful to maintain over time as clusters shrink, grow, 
or data shape changes. It'd be a lot better if the L1 did not leak so you don't 
have to so finely tune the L1.

I haven't fully figured out where the leak comes from, but I think it's related 
to compactions. Perhaps the INDEX blocks are not being evicted as hfiles are 
compacted away. The leak is very linear over time in our experience.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HBASE-25516) [JDK17] reflective access Field.class.getDeclaredField("modifiers") not supported

[Duo Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-25516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-25516.
---
Fix Version/s: 2.6.0
   2.4.17
   2.5.4
   Resolution: Fixed

CHerry-picked to branch-2.x.

> [JDK17] reflective access Field.class.getDeclaredField("modifiers") not 
> supported
> -
>
> Key: HBASE-25516
> URL: https://issues.apache.org/jira/browse/HBASE-25516
> Project: HBase
>  Issue Type: Sub-task
>  Components: Filesystem Integration
>Affects Versions: 2.3.3
> Environment: Windows 10, JavaSE11, pom dependencies:
> {code:java}
> 
>   org.apache.hbase
>   hbase-testing-util
>   2.3.3
> 
> 
>   junit
>   junit
>   4.12
> {code}
>Reporter: Leon Bein
>Assignee: Wei-Chiu Chuang
>Priority: Major
>  Labels: jdk11
> Fix For: 2.6.0, 2.4.17, 2.5.4, 3.0.0-alpha-2
>
>
> The reflective access
> {code:java}
> Field.class.getDeclaredField("modifiers")
> {code}
> in HFileSystem.java:334 leads to a warning (and probably error?):
>  
> {code:java}
> java.lang.NoSuchFieldException: modifiers
>   at java.base/java.lang.Class.getDeclaredField(Class.java:2417)
>   at 
> org.apache.hadoop.hbase.fs.HFileSystem.addLocationsOrderInterceptor(HFileSystem.java:334)
>   at 
> org.apache.hadoop.hbase.fs.HFileSystem.addLocationsOrderInterceptor(HFileSystem.java:291)
>   at org.apache.hadoop.hbase.fs.HFileSystem.(HFileSystem.java:96)
>   at org.apache.hadoop.hbase.fs.HFileSystem.get(HFileSystem.java:465)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.getTestFileSystem(HBaseTestingUtility.java:3330)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.getNewDataTestDirOnTestFS(HBaseTestingUtility.java:565)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.setupDataTestDirOnTestFS(HBaseTestingUtility.java:554)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.getDataTestDirOnTestFS(HBaseTestingUtility.java:527)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.getDefaultRootDirPath(HBaseTestingUtility.java:1415)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.createRootDir(HBaseTestingUtility.java:1446)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1157)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1144)
>   at foo.Main.main(Main.java:11)
> {code}
> when running the following code:
>  
> {code:java}
> public static void main(String[] args) throws Exception {
> HBaseTestingUtility utility = new 
> HBaseTestingUtility(HBaseConfiguration.create());
> 
> utility.startMiniCluster(StartMiniClusterOption.builder().numRegionServers(3).build());
> }{code}
> From my knowledge this results from the more restrictive reflection 
> protection of java.base classes in the newer java versions.
>  
> Related to HBASE-22972
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (HBASE-25516) [JDK17] reflective access Field.class.getDeclaredField("modifiers") not supported

[Duo Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-25516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang reopened HBASE-25516:
---

> [JDK17] reflective access Field.class.getDeclaredField("modifiers") not 
> supported
> -
>
> Key: HBASE-25516
> URL: https://issues.apache.org/jira/browse/HBASE-25516
> Project: HBase
>  Issue Type: Sub-task
>  Components: Filesystem Integration
>Affects Versions: 2.3.3
> Environment: Windows 10, JavaSE11, pom dependencies:
> {code:java}
> 
>   org.apache.hbase
>   hbase-testing-util
>   2.3.3
> 
> 
>   junit
>   junit
>   4.12
> {code}
>Reporter: Leon Bein
>Assignee: Wei-Chiu Chuang
>Priority: Major
>  Labels: jdk11
> Fix For: 3.0.0-alpha-2
>
>
> The reflective access
> {code:java}
> Field.class.getDeclaredField("modifiers")
> {code}
> in HFileSystem.java:334 leads to a warning (and probably error?):
>  
> {code:java}
> java.lang.NoSuchFieldException: modifiers
>   at java.base/java.lang.Class.getDeclaredField(Class.java:2417)
>   at 
> org.apache.hadoop.hbase.fs.HFileSystem.addLocationsOrderInterceptor(HFileSystem.java:334)
>   at 
> org.apache.hadoop.hbase.fs.HFileSystem.addLocationsOrderInterceptor(HFileSystem.java:291)
>   at org.apache.hadoop.hbase.fs.HFileSystem.(HFileSystem.java:96)
>   at org.apache.hadoop.hbase.fs.HFileSystem.get(HFileSystem.java:465)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.getTestFileSystem(HBaseTestingUtility.java:3330)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.getNewDataTestDirOnTestFS(HBaseTestingUtility.java:565)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.setupDataTestDirOnTestFS(HBaseTestingUtility.java:554)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.getDataTestDirOnTestFS(HBaseTestingUtility.java:527)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.getDefaultRootDirPath(HBaseTestingUtility.java:1415)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.createRootDir(HBaseTestingUtility.java:1446)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1157)
>   at 
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1144)
>   at foo.Main.main(Main.java:11)
> {code}
> when running the following code:
>  
> {code:java}
> public static void main(String[] args) throws Exception {
> HBaseTestingUtility utility = new 
> HBaseTestingUtility(HBaseConfiguration.create());
> 
> utility.startMiniCluster(StartMiniClusterOption.builder().numRegionServers(3).build());
> }{code}
> From my knowledge this results from the more restrictive reflection 
> protection of java.base classes in the newer java versions.
>  
> Related to HBASE-22972
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [HELP WANTED] Committers please help reviewing HBASE-27583

[Bryan Beaudreault]

Done, thanks for handling this Duo.

On Tue, Jan 24, 2023 at 9:27 AM 张铎(Duo Zhang)  wrote:

> The output of our protoc check eats a lot of space on the jenkins
> server and the infra team has already contacted the PMC about this
> issue.
>
> I've opened a PR to remove the -X option when running mvn command line
> in protoc check. Please help reviewing it and let's see if this could
> fix the space issue.
>
> https://github.com/apache/hbase/pull/4988
> 
>
> Thanks.
>

[HELP WANTED] Committers please help reviewing HBASE-27583

[Duo Zhang]

The output of our protoc check eats a lot of space on the jenkins
server and the infra team has already contacted the PMC about this
issue.

I've opened a PR to remove the -X option when running mvn command line
in protoc check. Please help reviewing it and let's see if this could
fix the space issue.

https://github.com/apache/hbase/pull/4988

Thanks.

[jira] [Resolved] (HBASE-27542) Remove unneeded distcp log cleanup after incremental backups

[Bryan Beaudreault (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-27542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Beaudreault resolved HBASE-27542.
---
Resolution: Fixed

> Remove unneeded distcp log cleanup after incremental backups
> 
>
> Key: HBASE-27542
> URL: https://issues.apache.org/jira/browse/HBASE-27542
> Project: HBase
>  Issue Type: Improvement
>  Components: backuprestore
>Affects Versions: 3.0.0-alpha-3
>Reporter: Jarryd Lee
>Assignee: Jarryd Lee
>Priority: Minor
> Fix For: 2.6.0, 3.0.0-alpha-4
>
>
> During the completion step of incremental backups, the 
> [TableBackupClient|https://github.com/apache/hbase/blob/2c3abae18aa35e2693b64b143316817d4569d0c3/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/TableBackupClient.java#L343-L355]
>  ensures distcp logs are cleaned up. However, 
> [DistCp|https://github.com/apache/hadoop/blob/b87c0ea7ebde3edc312dcc8938809610a914df7f/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java#L465-L476]
>  already ensures that the metafolder, where the distcp logs are stored, is 
> cleaned up via the 
> [CopyCommitter|https://github.com/apache/hadoop/blob/b743d56eb4bf350448bd315540fde4f029175082/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java#L189].
> Additionally, when running in an external yarn cluster external to hbase, 
> this causes issues as it assumes a common filesystem.
> The TableBackupClient cleanup method should be able to be safely removed.
> e: After digging in to this even more, the `_distcp_log` path used in the 
> `TableBackupClient` is not actually the correct path used by distcp.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HBASE-26300) Skip archived master wals during incremental backups

[Bryan Beaudreault (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Beaudreault resolved HBASE-26300.
---
Resolution: Fixed

> Skip archived master wals during incremental backups
> 
>
> Key: HBASE-26300
> URL: https://issues.apache.org/jira/browse/HBASE-26300
> Project: HBase
>  Issue Type: Bug
>  Components: backuprestore
>Reporter: Bryan Beaudreault
>Assignee: Jarryd Lee
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-4
>
>
> I've been reading through the incremental backup implementation in master 
> branch to see how it handled some scenarios our own internal incremental 
> backup process has to handle. One such failure we recently encountered as 
> part of our ongoing hbase2 upgrade is the new $masterlocalwal$ suffixed files 
> in the oldWALs dir. Our parsing of the WAL files assumed that the last part 
> of the file name would be a timestamp, which is not the case for these 
> MasterRegion WALs.
> I see [IncrementalBackupManager excludes 
> ProcV2Wals|https://github.com/apache/hbase/blob/master/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalBackupManager.java#L104-L117],
>  but I think that was replaced in 
> https://issues.apache.org/jira/browse/HBASE-24408 with a MasterRegion. The 
> new MasterRegion uses normal WALs, but archives them with a suffix 
> "$masterlocalwal$".
> I believe this would fail [around line 222 of 
> IncrementalBackupManager|https://github.com/apache/hbase/blob/master/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalBackupManager.java#L222],
>  because 
> [BackupUtils.getCreationTime|https://github.com/apache/hbase/blob/master/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/util/BackupUtils.java#L383-L390]
>  similarly expects the file names to end with a timestamp.
> Unfortunately I am not set up to run master branch or test the backup/restore 
> functionality, but I wanted to log this because I happened to stumble upon it.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HBASE-27541) Backups should be able to be restored to a separate filesystem

[Bryan Beaudreault (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-27541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Beaudreault resolved HBASE-27541.
---
Resolution: Fixed

> Backups should be able to be restored to a separate filesystem
> --
>
> Key: HBASE-27541
> URL: https://issues.apache.org/jira/browse/HBASE-27541
> Project: HBase
>  Issue Type: Improvement
>  Components: backuprestore
>Affects Versions: 3.0.0-alpha-3
>Reporter: Jarryd Lee
>Assignee: Jarryd Lee
>Priority: Minor
> Fix For: 2.6.0, 3.0.0-alpha-4
>
>
> Currently it's not possible to set the hfile output path for a given restore 
> job. This prevents us from running restore jobs in external yarn clusters.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (HBASE-27542) Remove unneeded distcp log cleanup after incremental backups

[Bryan Beaudreault (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-27542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Beaudreault reopened HBASE-27542:
---
  Assignee: Jarryd Lee

Reopening for port to branch-2

> Remove unneeded distcp log cleanup after incremental backups
> 
>
> Key: HBASE-27542
> URL: https://issues.apache.org/jira/browse/HBASE-27542
> Project: HBase
>  Issue Type: Improvement
>  Components: backuprestore
>Affects Versions: 3.0.0-alpha-3
>Reporter: Jarryd Lee
>Assignee: Jarryd Lee
>Priority: Minor
> Fix For: 3.0.0-alpha-4
>
>
> During the completion step of incremental backups, the 
> [TableBackupClient|https://github.com/apache/hbase/blob/2c3abae18aa35e2693b64b143316817d4569d0c3/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/TableBackupClient.java#L343-L355]
>  ensures distcp logs are cleaned up. However, 
> [DistCp|https://github.com/apache/hadoop/blob/b87c0ea7ebde3edc312dcc8938809610a914df7f/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java#L465-L476]
>  already ensures that the metafolder, where the distcp logs are stored, is 
> cleaned up via the 
> [CopyCommitter|https://github.com/apache/hadoop/blob/b743d56eb4bf350448bd315540fde4f029175082/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java#L189].
> Additionally, when running in an external yarn cluster external to hbase, 
> this causes issues as it assumes a common filesystem.
> The TableBackupClient cleanup method should be able to be safely removed.
> e: After digging in to this even more, the `_distcp_log` path used in the 
> `TableBackupClient` is not actually the correct path used by distcp.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (HBASE-27541) Backups should be able to be restored to a separate filesystem

[Bryan Beaudreault (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-27541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Beaudreault reopened HBASE-27541:
---
  Assignee: Jarryd Lee

Reopening for port to branch-2

> Backups should be able to be restored to a separate filesystem
> --
>
> Key: HBASE-27541
> URL: https://issues.apache.org/jira/browse/HBASE-27541
> Project: HBase
>  Issue Type: Improvement
>  Components: backuprestore
>Affects Versions: 3.0.0-alpha-3
>Reporter: Jarryd Lee
>Assignee: Jarryd Lee
>Priority: Minor
> Fix For: 3.0.0-alpha-4
>
>
> Currently it's not possible to set the hfile output path for a given restore 
> job. This prevents us from running restore jobs in external yarn clusters.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (HBASE-26300) Skip archived master wals during incremental backups

[Bryan Beaudreault (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Beaudreault reopened HBASE-26300:
---
  Assignee: Jarryd Lee

Reopening for port to branch-2

> Skip archived master wals during incremental backups
> 
>
> Key: HBASE-26300
> URL: https://issues.apache.org/jira/browse/HBASE-26300
> Project: HBase
>  Issue Type: Bug
>Reporter: Bryan Beaudreault
>Assignee: Jarryd Lee
>Priority: Major
> Fix For: 3.0.0-alpha-4
>
>
> I've been reading through the incremental backup implementation in master 
> branch to see how it handled some scenarios our own internal incremental 
> backup process has to handle. One such failure we recently encountered as 
> part of our ongoing hbase2 upgrade is the new $masterlocalwal$ suffixed files 
> in the oldWALs dir. Our parsing of the WAL files assumed that the last part 
> of the file name would be a timestamp, which is not the case for these 
> MasterRegion WALs.
> I see [IncrementalBackupManager excludes 
> ProcV2Wals|https://github.com/apache/hbase/blob/master/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalBackupManager.java#L104-L117],
>  but I think that was replaced in 
> https://issues.apache.org/jira/browse/HBASE-24408 with a MasterRegion. The 
> new MasterRegion uses normal WALs, but archives them with a suffix 
> "$masterlocalwal$".
> I believe this would fail [around line 222 of 
> IncrementalBackupManager|https://github.com/apache/hbase/blob/master/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalBackupManager.java#L222],
>  because 
> [BackupUtils.getCreationTime|https://github.com/apache/hbase/blob/master/hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/util/BackupUtils.java#L383-L390]
>  similarly expects the file names to end with a timestamp.
> Unfortunately I am not set up to run master branch or test the backup/restore 
> functionality, but I wanted to log this because I happened to stumble upon it.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)