Re: [DISCUSS] Replace log4j with reload4j for branch-2.x
Oh, shit, this would introduce more problems than benefits in the real world... If some projects depend on both log4j and slf4j-log4j12, once they upgrade the version of slf4j to 1.7.35 without changing log4j to reload4j, log4j and reload4j will both be included in the final artifact... Anyway, it does not affect us, let's just use slf4j-reload4j directly... Wei-Chiu Chuang 于2022年1月28日周五 12:34写道： > https://www.slf4j.org/news.html > BTW, a new slf4j-log4j12 release is made where it switches dependency on > log4j12 to reload4j: > https://mvnrepository.com/artifact/org.slf4j/slf4j-log4j12/1.7.35 > > This should hopefully make the migration slightly easier. > > On Tue, Jan 25, 2022 at 5:16 PM 张铎(Duo Zhang) > wrote: > > > I would prefer we have LOG4J2-3341 first before releasing any 2.x > releases > > with log4j2. I pinged the log4j community once but no response yet. Will > > try to ping again soon. > > > > Andrew Purtell 于2022年1月25日周二 10:09写道： > > > > > Reload4J is a great option when we really should maintain fairly strict > > > compatibility (patch releases) and less so when not (minors, majors). > > > > > > We haven’t released 2.5.0 yet. My fault, mainly... It’s been lagging. > > > Perhaps we could backport Duo’s work from 3.0.0-alpha into branch-2.5 / > > > 2.5.0? This would be a minor release, so, while some operational > > > compatibility breaks would be somewhat surprising, it could be more > > > understandable. > > > > > > > On Jan 24, 2022, at 5:00 PM, Zach York wrote: > > > > > > > > +1 on the original discussion > > > > > > > > I think that all makes sense, we can't know whether one dependency is > > > more > > > > vulnerable/going to be more work to maintain or not. However, I think > > > > perhaps the more interesting question is whether we should be okay > with > > > > using EOL dependencies in any active release line. CVEs happen... I > > think > > > > it's important to be able to react quickly. By depending on any EOL > > code > > > in > > > > our active release lines, we severely limit our ability to quickly > deal > > > > with CVEs. I understand it's not always easy (the backwards > > > incompatibility > > > > of log4j2 and the properties files are good examples), but it's also > > been > > > > 6+ years since log4j 1.x has become EOL. > > > > > > > >> On Fri, Jan 21, 2022 at 11:47 AM Andrew Purtell < > apurt...@apache.org> > > > wrote: > > > >> > > > >> I will offer a guess as answer to the question originally proposed, > > > which > > > >> is, no the Logging PMC is not going to behave in a cooperative > manner > > to > > > >> Reload4J. The best we can hope for, and I personally doubt it, is > they > > > will > > > >> not be actively hostile. Decisions made by Apache PMC can > > unfortunately > > > be > > > >> made on the basis of years-long running arguments and personality > > > >> conflicts, and Logging is unfortunately one of them. Just my > opinion. > > > You > > > >> won't change it. Fortunately that is neither here nor there. We > aren't > > > here > > > >> to judge up front if Reload4J can respond adequately to hypothetical > > > future > > > >> security issues. That is not knowable and remains to be seen. It > also > > > >> remains to be seen if Log4J 2 is going to respond adequately to > future > > > >> security issues. Or Netty. Or Jersey. Or Jackson. Or any of our > other > > > risky > > > >> third party dependencies (as measured by having "interesting" CVEs). > > > What > > > >> we can do is look at the current track record, which has been > adequate > > > so > > > >> far. > > > >> > > > >> > > > >> On Fri, Jan 21, 2022 at 11:35 AM Wei-Chiu Chuang > > > >> wrote: > > > >> > > > >>> The reload4j is maintained by one of the Apache Logging PMC. And a > > > >> release > > > >>> was made to address the latest log4j 1.x CVEs announced only a day > > > after. > > > >>> > > > >>> There is a thread in the private@logging that mentioned the “new” > > > >> log4j1.x > > > >>> CVEs were actually not new. They were announced years before for > 2.x > > > >> which > > > >>> happens to also impact 1.x. But because 1.x was considered EOL, > they > > > >> didn’t > > > >>> bother to mention 1.x were affected. It is only now that 1.x’s > > getting > > > >>> interest that they did this. > > > >>> > > > >>> Sean Busbey 於 2022年1月22日 週六，上午2:47寫道： > > > >>> > > > It's relevant to what kind of mitigation this is. The > effectiveness > > of > > > reload4j to deal with "the critical CVEs of log4j 1" is limited by > > how > > > likely it is that they know about them. > > > > > > Otherwise at the next CVE we're back in the same place where > > > downstream > > > users aren't meaningfully more protected. And in that case perhaps > > we > > > >>> would > > > do better for our users by e.g. putting more emphasis upgrading > > across > > > >>> our > > > releases or providing breaking changes to get the pain over with. > > > > > > On Fri, Jan 21, 2022 at
[jira] [Resolved] (HBASE-25918) Upgrade hbase-thirdparty dependency to 3.5.1
[ https://issues.apache.org/jira/browse/HBASE-25918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-25918. - Resolution: Fixed > Upgrade hbase-thirdparty dependency to 3.5.1 > > > Key: HBASE-25918 > URL: https://issues.apache.org/jira/browse/HBASE-25918 > Project: HBase > Issue Type: Task > Components: dependencies >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Critical > Fix For: 2.5.0, 2.4.5, 3.0.0-alpha-1 > > > Recently we have fixed multiple CVEs from jetty & netty as part of > HBASE-25728 & HBASE-25746. This Jira is to upgrade hbase-thirdparty jar > version. -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [DISCUSS] Replace log4j with reload4j for branch-2.x
https://www.slf4j.org/news.html BTW, a new slf4j-log4j12 release is made where it switches dependency on log4j12 to reload4j: https://mvnrepository.com/artifact/org.slf4j/slf4j-log4j12/1.7.35 This should hopefully make the migration slightly easier. On Tue, Jan 25, 2022 at 5:16 PM 张铎(Duo Zhang) wrote: > I would prefer we have LOG4J2-3341 first before releasing any 2.x releases > with log4j2. I pinged the log4j community once but no response yet. Will > try to ping again soon. > > Andrew Purtell 于2022年1月25日周二 10:09写道： > > > Reload4J is a great option when we really should maintain fairly strict > > compatibility (patch releases) and less so when not (minors, majors). > > > > We haven’t released 2.5.0 yet. My fault, mainly... It’s been lagging. > > Perhaps we could backport Duo’s work from 3.0.0-alpha into branch-2.5 / > > 2.5.0? This would be a minor release, so, while some operational > > compatibility breaks would be somewhat surprising, it could be more > > understandable. > > > > > On Jan 24, 2022, at 5:00 PM, Zach York wrote: > > > > > > +1 on the original discussion > > > > > > I think that all makes sense, we can't know whether one dependency is > > more > > > vulnerable/going to be more work to maintain or not. However, I think > > > perhaps the more interesting question is whether we should be okay with > > > using EOL dependencies in any active release line. CVEs happen... I > think > > > it's important to be able to react quickly. By depending on any EOL > code > > in > > > our active release lines, we severely limit our ability to quickly deal > > > with CVEs. I understand it's not always easy (the backwards > > incompatibility > > > of log4j2 and the properties files are good examples), but it's also > been > > > 6+ years since log4j 1.x has become EOL. > > > > > >> On Fri, Jan 21, 2022 at 11:47 AM Andrew Purtell > > wrote: > > >> > > >> I will offer a guess as answer to the question originally proposed, > > which > > >> is, no the Logging PMC is not going to behave in a cooperative manner > to > > >> Reload4J. The best we can hope for, and I personally doubt it, is they > > will > > >> not be actively hostile. Decisions made by Apache PMC can > unfortunately > > be > > >> made on the basis of years-long running arguments and personality > > >> conflicts, and Logging is unfortunately one of them. Just my opinion. > > You > > >> won't change it. Fortunately that is neither here nor there. We aren't > > here > > >> to judge up front if Reload4J can respond adequately to hypothetical > > future > > >> security issues. That is not knowable and remains to be seen. It also > > >> remains to be seen if Log4J 2 is going to respond adequately to future > > >> security issues. Or Netty. Or Jersey. Or Jackson. Or any of our other > > risky > > >> third party dependencies (as measured by having "interesting" CVEs). > > What > > >> we can do is look at the current track record, which has been adequate > > so > > >> far. > > >> > > >> > > >> On Fri, Jan 21, 2022 at 11:35 AM Wei-Chiu Chuang > > >> wrote: > > >> > > >>> The reload4j is maintained by one of the Apache Logging PMC. And a > > >> release > > >>> was made to address the latest log4j 1.x CVEs announced only a day > > after. > > >>> > > >>> There is a thread in the private@logging that mentioned the “new” > > >> log4j1.x > > >>> CVEs were actually not new. They were announced years before for 2.x > > >> which > > >>> happens to also impact 1.x. But because 1.x was considered EOL, they > > >> didn’t > > >>> bother to mention 1.x were affected. It is only now that 1.x’s > getting > > >>> interest that they did this. > > >>> > > >>> Sean Busbey 於 2022年1月22日 週六，上午2:47寫道： > > >>> > > It's relevant to what kind of mitigation this is. The effectiveness > of > > reload4j to deal with "the critical CVEs of log4j 1" is limited by > how > > likely it is that they know about them. > > > > Otherwise at the next CVE we're back in the same place where > > downstream > > users aren't meaningfully more protected. And in that case perhaps > we > > >>> would > > do better for our users by e.g. putting more emphasis upgrading > across > > >>> our > > releases or providing breaking changes to get the pain over with. > > > > On Fri, Jan 21, 2022 at 11:03 AM Andrew Purtell < > > >>> andrew.purt...@gmail.com> > > wrote: > > > > > That does not seem germane to this discussion. We don’t investigate > > >> and > > > attempt to manage the security reporting arrangement of any of our > > >>> other > > > third party dependencies. > > > > > >> On Jan 21, 2022, at 7:59 AM, Sean Busbey > > >> wrote: > > >> > > >> Has anyone asked the ASF Logging PMC if they'll forward security > > reports > > >> against log4j 1 to the reload4j project? > > >> > > >>> On Fri, Jan 21, 2022 at 3:33 AM Pankaj Kumar < > > >>> pankajku...@apache.org> > > > wrote: > > >>> > > >>>
hbase-thirdparty 3.5.1 contains unshaded classes
Please see HBASE-26717. I recommend we fix and release a 3.5.2 to correct it. -- Best regards, Andrew Unrest, ignorance distilled, nihilistic imbeciles - It's what we’ve earned Welcome, apocalypse, what’s taken you so long? Bring us the fitting end that we’ve been counting on - A23, Welcome, Apocalypse
Re: [DISCUSS] Replace log4j with reload4j for branch-2.x
I think we should replace log4j:log4j* with ch.qos.reload4j:reload4j:1.2.18.3 in every relevant POM. On Thu, Jan 27, 2022 at 5:45 PM Wei-Chiu Chuang wrote: > What's the plan for other repos? > hbase-filesystem, hbase-connectors, hbase-operator-tools depends on log4j > 1.x > > Even hbase-thirdparty has transitive dependency on log4j 1.x > > On Tue, Jan 25, 2022 at 5:16 PM 张铎(Duo Zhang) > wrote: > > > I would prefer we have LOG4J2-3341 first before releasing any 2.x > releases > > with log4j2. I pinged the log4j community once but no response yet. Will > > try to ping again soon. > > > > Andrew Purtell 于2022年1月25日周二 10:09写道： > > > > > Reload4J is a great option when we really should maintain fairly strict > > > compatibility (patch releases) and less so when not (minors, majors). > > > > > > We haven’t released 2.5.0 yet. My fault, mainly... It’s been lagging. > > > Perhaps we could backport Duo’s work from 3.0.0-alpha into branch-2.5 / > > > 2.5.0? This would be a minor release, so, while some operational > > > compatibility breaks would be somewhat surprising, it could be more > > > understandable. > > > > > > > On Jan 24, 2022, at 5:00 PM, Zach York wrote: > > > > > > > > +1 on the original discussion > > > > > > > > I think that all makes sense, we can't know whether one dependency is > > > more > > > > vulnerable/going to be more work to maintain or not. However, I think > > > > perhaps the more interesting question is whether we should be okay > with > > > > using EOL dependencies in any active release line. CVEs happen... I > > think > > > > it's important to be able to react quickly. By depending on any EOL > > code > > > in > > > > our active release lines, we severely limit our ability to quickly > deal > > > > with CVEs. I understand it's not always easy (the backwards > > > incompatibility > > > > of log4j2 and the properties files are good examples), but it's also > > been > > > > 6+ years since log4j 1.x has become EOL. > > > > > > > >> On Fri, Jan 21, 2022 at 11:47 AM Andrew Purtell < > apurt...@apache.org> > > > wrote: > > > >> > > > >> I will offer a guess as answer to the question originally proposed, > > > which > > > >> is, no the Logging PMC is not going to behave in a cooperative > manner > > to > > > >> Reload4J. The best we can hope for, and I personally doubt it, is > they > > > will > > > >> not be actively hostile. Decisions made by Apache PMC can > > unfortunately > > > be > > > >> made on the basis of years-long running arguments and personality > > > >> conflicts, and Logging is unfortunately one of them. Just my > opinion. > > > You > > > >> won't change it. Fortunately that is neither here nor there. We > aren't > > > here > > > >> to judge up front if Reload4J can respond adequately to hypothetical > > > future > > > >> security issues. That is not knowable and remains to be seen. It > also > > > >> remains to be seen if Log4J 2 is going to respond adequately to > future > > > >> security issues. Or Netty. Or Jersey. Or Jackson. Or any of our > other > > > risky > > > >> third party dependencies (as measured by having "interesting" CVEs). > > > What > > > >> we can do is look at the current track record, which has been > adequate > > > so > > > >> far. > > > >> > > > >> > > > >> On Fri, Jan 21, 2022 at 11:35 AM Wei-Chiu Chuang > > > >> wrote: > > > >> > > > >>> The reload4j is maintained by one of the Apache Logging PMC. And a > > > >> release > > > >>> was made to address the latest log4j 1.x CVEs announced only a day > > > after. > > > >>> > > > >>> There is a thread in the private@logging that mentioned the “new” > > > >> log4j1.x > > > >>> CVEs were actually not new. They were announced years before for > 2.x > > > >> which > > > >>> happens to also impact 1.x. But because 1.x was considered EOL, > they > > > >> didn’t > > > >>> bother to mention 1.x were affected. It is only now that 1.x’s > > getting > > > >>> interest that they did this. > > > >>> > > > >>> Sean Busbey 於 2022年1月22日 週六，上午2:47寫道： > > > >>> > > > It's relevant to what kind of mitigation this is. The > effectiveness > > of > > > reload4j to deal with "the critical CVEs of log4j 1" is limited by > > how > > > likely it is that they know about them. > > > > > > Otherwise at the next CVE we're back in the same place where > > > downstream > > > users aren't meaningfully more protected. And in that case perhaps > > we > > > >>> would > > > do better for our users by e.g. putting more emphasis upgrading > > across > > > >>> our > > > releases or providing breaking changes to get the pain over with. > > > > > > On Fri, Jan 21, 2022 at 11:03 AM Andrew Purtell < > > > >>> andrew.purt...@gmail.com> > > > wrote: > > > > > > > That does not seem germane to this discussion. We don’t > investigate > > > >> and > > > > attempt to manage the security reporting arrangement of any of > our > > > >>> other > > > > third party dependencies. > > >
Re: [DISCUSS] Replace log4j with reload4j for branch-2.x
What's the plan for other repos? hbase-filesystem, hbase-connectors, hbase-operator-tools depends on log4j 1.x Even hbase-thirdparty has transitive dependency on log4j 1.x On Tue, Jan 25, 2022 at 5:16 PM 张铎(Duo Zhang) wrote: > I would prefer we have LOG4J2-3341 first before releasing any 2.x releases > with log4j2. I pinged the log4j community once but no response yet. Will > try to ping again soon. > > Andrew Purtell 于2022年1月25日周二 10:09写道： > > > Reload4J is a great option when we really should maintain fairly strict > > compatibility (patch releases) and less so when not (minors, majors). > > > > We haven’t released 2.5.0 yet. My fault, mainly... It’s been lagging. > > Perhaps we could backport Duo’s work from 3.0.0-alpha into branch-2.5 / > > 2.5.0? This would be a minor release, so, while some operational > > compatibility breaks would be somewhat surprising, it could be more > > understandable. > > > > > On Jan 24, 2022, at 5:00 PM, Zach York wrote: > > > > > > +1 on the original discussion > > > > > > I think that all makes sense, we can't know whether one dependency is > > more > > > vulnerable/going to be more work to maintain or not. However, I think > > > perhaps the more interesting question is whether we should be okay with > > > using EOL dependencies in any active release line. CVEs happen... I > think > > > it's important to be able to react quickly. By depending on any EOL > code > > in > > > our active release lines, we severely limit our ability to quickly deal > > > with CVEs. I understand it's not always easy (the backwards > > incompatibility > > > of log4j2 and the properties files are good examples), but it's also > been > > > 6+ years since log4j 1.x has become EOL. > > > > > >> On Fri, Jan 21, 2022 at 11:47 AM Andrew Purtell > > wrote: > > >> > > >> I will offer a guess as answer to the question originally proposed, > > which > > >> is, no the Logging PMC is not going to behave in a cooperative manner > to > > >> Reload4J. The best we can hope for, and I personally doubt it, is they > > will > > >> not be actively hostile. Decisions made by Apache PMC can > unfortunately > > be > > >> made on the basis of years-long running arguments and personality > > >> conflicts, and Logging is unfortunately one of them. Just my opinion. > > You > > >> won't change it. Fortunately that is neither here nor there. We aren't > > here > > >> to judge up front if Reload4J can respond adequately to hypothetical > > future > > >> security issues. That is not knowable and remains to be seen. It also > > >> remains to be seen if Log4J 2 is going to respond adequately to future > > >> security issues. Or Netty. Or Jersey. Or Jackson. Or any of our other > > risky > > >> third party dependencies (as measured by having "interesting" CVEs). > > What > > >> we can do is look at the current track record, which has been adequate > > so > > >> far. > > >> > > >> > > >> On Fri, Jan 21, 2022 at 11:35 AM Wei-Chiu Chuang > > >> wrote: > > >> > > >>> The reload4j is maintained by one of the Apache Logging PMC. And a > > >> release > > >>> was made to address the latest log4j 1.x CVEs announced only a day > > after. > > >>> > > >>> There is a thread in the private@logging that mentioned the “new” > > >> log4j1.x > > >>> CVEs were actually not new. They were announced years before for 2.x > > >> which > > >>> happens to also impact 1.x. But because 1.x was considered EOL, they > > >> didn’t > > >>> bother to mention 1.x were affected. It is only now that 1.x’s > getting > > >>> interest that they did this. > > >>> > > >>> Sean Busbey 於 2022年1月22日 週六，上午2:47寫道： > > >>> > > It's relevant to what kind of mitigation this is. The effectiveness > of > > reload4j to deal with "the critical CVEs of log4j 1" is limited by > how > > likely it is that they know about them. > > > > Otherwise at the next CVE we're back in the same place where > > downstream > > users aren't meaningfully more protected. And in that case perhaps > we > > >>> would > > do better for our users by e.g. putting more emphasis upgrading > across > > >>> our > > releases or providing breaking changes to get the pain over with. > > > > On Fri, Jan 21, 2022 at 11:03 AM Andrew Purtell < > > >>> andrew.purt...@gmail.com> > > wrote: > > > > > That does not seem germane to this discussion. We don’t investigate > > >> and > > > attempt to manage the security reporting arrangement of any of our > > >>> other > > > third party dependencies. > > > > > >> On Jan 21, 2022, at 7:59 AM, Sean Busbey > > >> wrote: > > >> > > >> Has anyone asked the ASF Logging PMC if they'll forward security > > reports > > >> against log4j 1 to the reload4j project? > > >> > > >>> On Fri, Jan 21, 2022 at 3:33 AM Pankaj Kumar < > > >>> pankajku...@apache.org> > > > wrote: > > >>> > > >>> +1 for reload4j. > > >>> > > >>> Regards, > > >>> Pankaj > > >
[jira] [Reopened] (HBASE-25918) Upgrade hbase-thirdparty dependency to 3.5.1
[ https://issues.apache.org/jira/browse/HBASE-25918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell reopened HBASE-25918: - Reopening, see HBASE-26717 > Upgrade hbase-thirdparty dependency to 3.5.1 > > > Key: HBASE-25918 > URL: https://issues.apache.org/jira/browse/HBASE-25918 > Project: HBase > Issue Type: Task > Components: dependencies >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.5 > > > Recently we have fixed multiple CVEs from jetty & netty as part of > HBASE-25728 & HBASE-25746. This Jira is to upgrade hbase-thirdparty jar > version. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26720) ExportSnapshot should validate the source snapshot before copying files
David Manning created HBASE-26720:
-
Summary: ExportSnapshot should validate the source snapshot before
copying files
Key: HBASE-26720
URL: https://issues.apache.org/jira/browse/HBASE-26720
Project: HBase
Issue Type: Improvement
Components: snapshots
Affects Versions: 2.0.0, 3.0.0-alpha-1, 1.0.0, 0.99.0
Reporter: David Manning
Assignee: David Manning
Running {{ExportSnapshot}} with default parameters will copy the snapshot to a
target location, and then use {{verifySnapshot}} to validate the integrity of
the written snapshot. However, it is possible for the source snapshot to be
invalid which leads to an invalid exported snapshot.
We can validate the source snapshot before export.
By default, we can validate the source snapshot unless the
{{-no-target-verify}} parameter is set. We could also introduce a separate
parameter for {{-no-source-verify}} if an operator wanted to validate the
target but not validate the source for some reason, to provide some amount of
backwards compatibility if that scenario is important.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HBASE-26719) Remove 'patch' file added as part of commit for HBASE-25973
[ https://issues.apache.org/jira/browse/HBASE-26719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-26719. - Resolution: Fixed > Remove 'patch' file added as part of commit for HBASE-25973 > > > Key: HBASE-26719 > URL: https://issues.apache.org/jira/browse/HBASE-26719 > Project: HBase > Issue Type: Task >Affects Versions: 2.5.0, 2.6.0 >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Minor > Fix For: 2.5.0, 2.6.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26718) HFileArchiver can remove referenced StoreFiles from the archive
David Manning created HBASE-26718:
-
Summary: HFileArchiver can remove referenced StoreFiles from the
archive
Key: HBASE-26718
URL: https://issues.apache.org/jira/browse/HBASE-26718
Project: HBase
Issue Type: Bug
Components: Compaction, HFile, snapshots
Affects Versions: 2.0.0, 3.0.0-alpha-1, 1.0.0
Reporter: David Manning
Assignee: David Manning
There is a comment in {{HFileArchiver#resolveAndArchiveFile}}:
{code:java}
// if the file already exists in the archive, move that one to a timestamped
backup. This is a
// really, really unlikely situtation, where we get the same name for the
existing file, but
// is included just for that 1 in trillion chance.
{code}
In reality, we did encounter this frequently enough to cause problems. More
details will be included and linked in a separate issue.
But regardless of how we get into this situation, we can consider a different
approach to solving it. If we assume store files are immutable, and a store
file with the same name and location already exists in the archive, then it can
be safer to assume the file was already archived successfully, and react
accordingly.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26719) Remove 'patch' file added as part of commit for HBASE-25973
Andrew Kyle Purtell created HBASE-26719: --- Summary: Remove 'patch' file added as part of commit for HBASE-25973 Key: HBASE-26719 URL: https://issues.apache.org/jira/browse/HBASE-26719 Project: HBase Issue Type: Task Affects Versions: 2.4.9, 2.5.0 Reporter: Andrew Kyle Purtell Assignee: Andrew Kyle Purtell Fix For: 2.5.0, 2.4.10 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Reopened] (HBASE-26688) Threads shared EMPTY_RESULT may lead to unexpected client job down.
[
https://issues.apache.org/jira/browse/HBASE-26688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk reopened HBASE-26688:
--
> Threads shared EMPTY_RESULT may lead to unexpected client job down.
> ---
>
> Key: HBASE-26688
> URL: https://issues.apache.org/jira/browse/HBASE-26688
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Yutong Xiao
>Assignee: Yutong Xiao
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-3, 2.4.10
>
>
> Currently, we use a pre-created EMPTY_RESULT in the ProtoBuf.util to reduce
> the object creation. But these objects could be shared by multi client
> threads. The Result#cellScannerIndex related methods could throw confusing
> exception and make the client job down. Could refine the logic of these
> methods.
> The precreated objects in ProtoBufUtil.java:
> {code:java}
> private final static Cell[] EMPTY_CELL_ARRAY = new Cell[]{};
> private final static Result EMPTY_RESULT = Result.create(EMPTY_CELL_ARRAY);
> final static Result EMPTY_RESULT_EXISTS_TRUE = Result.create(null, true);
> final static Result EMPTY_RESULT_EXISTS_FALSE = Result.create(null, false);
> private final static Result EMPTY_RESULT_STALE =
> Result.create(EMPTY_CELL_ARRAY, null, true);
> {code}
> Result#advance
> {code:java}
> public boolean advance() {
> if (cells == null) return false;
> cellScannerIndex++;
> if (cellScannerIndex < this.cells.length) {
> return true;
> } else if (cellScannerIndex == this.cells.length) {
> return false;
> }
> // The index of EMPTY_RESULT could be incremented by multi threads and
> throw this exception.
> throw new NoSuchElementException("Cannot advance beyond the last cell");
> }
> {code}
> Result#current
> {code:java}
> public Cell current() {
> if (cells == null
> || cellScannerIndex == INITIAL_CELLSCANNER_INDEX
> || cellScannerIndex >= cells.length)
> return null;
>// Although it is almost impossible,
>// We can arrive here when the client threads share the common reused
> EMPTY_RESULT.
> return this.cells[cellScannerIndex];
> }
> {code}
> In this case, the client can easily got confusing exceptions even if they use
> different connections, tables in different threads.
> We should change the if condition cells == null to isEmpty() to avoid the
> client crashed from these exception.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26717) [branch-2.5] hbase-shaded-check-invariants check fails
Andrew Kyle Purtell created HBASE-26717: --- Summary: [branch-2.5] hbase-shaded-check-invariants check fails Key: HBASE-26717 URL: https://issues.apache.org/jira/browse/HBASE-26717 Project: HBase Issue Type: Task Affects Versions: 2.5.0 Environment: Apache Maven 3.6.3 Java version: 1.8.0_312, vendor: Andrew Purtell, runtime: /opt/java-8/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "5.10.0-11-arm64", arch: "aarch64", family: "unix" Reporter: Andrew Kyle Purtell Fix For: 2.5.0 Found artifact with unexpected contents: '/home/apurtell/src/hbase/hbase-shaded/hbase-shaded-mapreduce/target/hbase-shaded-mapreduce-2.5.0-SNAPSHOT.jar' com/sun/activation/ javax/activation/ javax/xml/bind/ -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Reopened] (HBASE-26472) Adhere to semantic conventions regarding table data operations
[ https://issues.apache.org/jira/browse/HBASE-26472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk reopened HBASE-26472: -- I found an issue with the branch-2 implementation emitting multiple spans for some of the batch operations. Reopening for addendum. > Adhere to semantic conventions regarding table data operations > -- > > Key: HBASE-26472 > URL: https://issues.apache.org/jira/browse/HBASE-26472 > Project: HBase > Issue Type: Sub-task >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-2, 2.6.0 > > > Follow the guidance outlined in > https://github.com/open-telemetry/opentelemetry-specification/blob/3e380e2/specification/trace/semantic_conventions/database.dm -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26716) NPE caused by converting uppercase hostname to lowercase in RegionMover
ruanhui created HBASE-26716:
---
Summary: NPE caused by converting uppercase hostname to lowercase
in RegionMover
Key: HBASE-26716
URL: https://issues.apache.org/jira/browse/HBASE-26716
Project: HBase
Issue Type: Improvement
Components: util
Affects Versions: 2.4.9
Reporter: ruanhui
In HBASE-19456, we introduced case-insensitivity feature in RegionMover and
converted uppercase hostnames to lowercase hostnames. But this maybe causes
that we can't get the rsgroup info of unloading server, because the addresses
in hbase case insensitive. This will make
org.apache.hadoop.hbase.util.TestRegionMoverWithRSGroupEnable fail.
2022-01-27T20:53:31,948 INFO [Time-limited test]
util.TestRegionMoverWithRSGroupEnable(127): Unloading {*}VM{*}-154-75-centos
2022-01-27T20:53:31,959 INFO
[RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=49232]
master.MasterRpcServices(3011): rsGroupInfo of {*}vm{*}-154-75-centos:39126 is
null
2022-01-27T20:53:31,961 INFO [pool-332-thread-1] util.RegionMover(419): rsgroup
of {*}vm{*}-154-75-centos:39126 is null
2022-01-27T20:53:31,961 ERROR [pool-332-thread-1] util.RegionMover(471): Error
while unloading regions
java.lang.NullPointerException: null
at
org.apache.hadoop.hbase.util.RegionMover.lambda$unloadRegions$3(RegionMover.java:421)
~[classes/:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_292]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_292]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_292]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26715) RegionServer should abort of rollWAL cannot complete in a timely manner
Bryan Beaudreault created HBASE-26715:
-
Summary: RegionServer should abort of rollWAL cannot complete in a
timely manner
Key: HBASE-26715
URL: https://issues.apache.org/jira/browse/HBASE-26715
Project: HBase
Issue Type: Bug
Reporter: Bryan Beaudreault
Ran into an issue on hbase 2.4.6, I think related to HBASE-26679. Individual
writes are blocking on SyncFuture, which never gets completed. Eventually (5m)
the writes timeout and fail. But the regionserver hung on like this basically
forever until I killed it about 14 hours later. While 26679 may fix the hang
bug, I think we should have additional protection against such zombie states.
In this case I think what happened is that the rollWAL was requested due to
failed appends, but it also hung forever. See the below stack trace:
{code:java}
Thread 240 (regionserver/host:60020.logRoller):
State: WAITING
Blocked count: 38
Waited count: 293
Waiting on java.util.concurrent.CompletableFuture$Signaller@13342c6d
Stack:
java.base@11.0.5/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.5/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
java.base@11.0.5/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796)
java.base@11.0.5/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128)
java.base@11.0.5/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823)
java.base@11.0.5/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998)
app//org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(AsyncProtobufLogWriter.java:189)
app//org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeMagicAndWALHeader(AsyncProtobufLogWriter.java:202)
app//org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:170)
app//org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:113)
app//org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:669)
app//org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:130)
app//org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:841)
app//org.apache.hadoop.hbase.wal.AbstractWALRoller$RollController.rollWal(AbstractWALRoller.java:268)
app//org.apache.hadoop.hbase.wal.AbstractWALRoller.run(AbstractWALRoller.java:187)
{code}
The wall roller thread was stuck on this wait seemingly forever, so it was
never able to roll the wal and get writes working again. I think we should add
a timeout here, and abort the regionserver if a WAL cannot be rolled in a
timely manner.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
