Re: [VOTE] HBase 2.4.14 release candidate (RC1) is available
Thanks Andrew. Update: With six (6) binding +1 votes, and two (2) nonbinding +1 vote, and no 0 or -1 votes, this vote passes. I have done most work for this release, once the download page is updated, announcement email will be sent out. Huaxiang Sent from my iPhone > On Aug 29, 2022, at 5:38 PM, Andrew Purtell wrote: > > +1 (binding) > >* Signature: ok >* Checksum : ok >* Rat check (1.8.0_332): ok > - mvn clean apache-rat:check >* Built from source (1.8.0_332): ok > - mvn clean install -DskipTests >* Unit tests pass (1.8.0_332): failed > - mvn package -P runAllTests -Dsurefire.rerunFailingTestsCount=3 > > TestSplitRegionWhileRSCrash times out. > > >> On Wed, Aug 24, 2022 at 8:59 AM Huaxiang Sun wrote: >> >> Please vote on this Apache hbase release candidate, >> >> hbase-2.4.14RC1 >> >> >> The VOTE will remain open for at least 72 hours. >> >> >> [ ] +1 Release this package as Apache hbase 2.4.14 >> >> [ ] -1 Do not release this package because ... >> >> >> The tag to be voted on is 2.4.14RC1: >> >> >> https://github.com/apache/hbase/tree/2.4.14RC1 >> >> >> This tag currently points to git reference >> >> >> 2e7d75a89271a7479b2f668c4db7a241be3f >> >> >> The release files, including signatures, digests, as well as CHANGES.md >> >> and RELEASENOTES.md included in this RC can be found at: >> >> >> https://dist.apache.org/repos/dist/dev/hbase/2.4.14RC1/ >> >> >> Maven artifacts are available in a staging repository at: >> >> >> https://repository.apache.org/content/repositories/orgapachehbase-1495/ >> >> >> Artifacts were signed with the 0x117C835E key which can be found in: >> >> >> https://downloads.apache.org/hbase/KEYS >> >> >> To learn more about Apache hbase, please see >> >> >> http://hbase.apache.org/ >> >> >> Thanks, >> >> Your HBase Release Manager >> > > > -- > Best regards, > Andrew > > Unrest, ignorance distilled, nihilistic imbeciles - >It's what we’ve earned > Welcome, apocalypse, what’s taken you so long? > Bring us the fitting end that we’ve been counting on > - A23, Welcome, Apocalypse
Re: [VOTE] HBase 2.4.14 release candidate (RC1) is available
+1 (binding) * Signature: ok * Checksum : ok * Rat check (1.8.0_332): ok - mvn clean apache-rat:check * Built from source (1.8.0_332): ok - mvn clean install -DskipTests * Unit tests pass (1.8.0_332): failed - mvn package -P runAllTests -Dsurefire.rerunFailingTestsCount=3 TestSplitRegionWhileRSCrash times out. On Wed, Aug 24, 2022 at 8:59 AM Huaxiang Sun wrote: > Please vote on this Apache hbase release candidate, > > hbase-2.4.14RC1 > > > The VOTE will remain open for at least 72 hours. > > > [ ] +1 Release this package as Apache hbase 2.4.14 > > [ ] -1 Do not release this package because ... > > > The tag to be voted on is 2.4.14RC1: > > > https://github.com/apache/hbase/tree/2.4.14RC1 > > > This tag currently points to git reference > > > 2e7d75a89271a7479b2f668c4db7a241be3f > > > The release files, including signatures, digests, as well as CHANGES.md > > and RELEASENOTES.md included in this RC can be found at: > > > https://dist.apache.org/repos/dist/dev/hbase/2.4.14RC1/ > > > Maven artifacts are available in a staging repository at: > > > https://repository.apache.org/content/repositories/orgapachehbase-1495/ > > > Artifacts were signed with the 0x117C835E key which can be found in: > > > https://downloads.apache.org/hbase/KEYS > > > To learn more about Apache hbase, please see > > > http://hbase.apache.org/ > > > Thanks, > > Your HBase Release Manager > -- Best regards, Andrew Unrest, ignorance distilled, nihilistic imbeciles - It's what we’ve earned Welcome, apocalypse, what’s taken you so long? Bring us the fitting end that we’ve been counting on - A23, Welcome, Apocalypse
Re: [VOTE] Release candidate for HBase 2.5.0 (RC1) is available
+1 (binding) * Signature: ok * Checksum : ok * Rat check (1.8.0_332): ok - mvn clean apache-rat:check * Built from source (1.8.0_332): ok - mvn clean install -DskipTests * Unit tests pass (1.8.0_332): failed - mvn package -P runAllTests -Dsurefire.rerunFailingTestsCount=3 TestQuotaAdmin is failing consistently. It's a known issue if I recall correctly and is in the flaky list on ci-hbase. TestClearRegionBlockCache and TestReplicationKillSlaveRSWithSeparateOldWALs were flaky. On Wed, Aug 24, 2022 at 2:53 AM Nick Dimiduk wrote: > Please vote on this Apache hbase release candidate, > hbase-2.5.0RC1 > > The VOTE will remain open for at least 72 hours. > > [ ] +1 Release this package as Apache hbase 2.5.0 > [ ] -1 Do not release this package because ... > > The tag to be voted on is 2.5.0RC1: > > https://github.com/apache/hbase/tree/2.5.0RC1 > > This tag currently points to git reference > > 2ecd8bd6d615ca49bfb329b3c0c126c80846d4ab > > The release files, including signatures, digests, as well as CHANGES.md > and RELEASENOTES.md included in this RC can be found at: > > https://dist.apache.org/repos/dist/dev/hbase/2.5.0RC1/ > > Maven artifacts are available in a staging repository at: > > https://repository.apache.org/content/repositories/orgapachehbase-1494/ > > Artifacts were signed with the 0x18567F39 key which can be found in: > > https://downloads.apache.org/hbase/KEYS > > To learn more about Apache hbase, please see > > http://hbase.apache.org/ > > Thanks, > Your HBase Release Manager > -- Best regards, Andrew Unrest, ignorance distilled, nihilistic imbeciles - It's what we’ve earned Welcome, apocalypse, what’s taken you so long? Bring us the fitting end that we’ve been counting on - A23, Welcome, Apocalypse
[jira] [Resolved] (HBASE-27345) Add 2.4.14 to the downloads page
[ https://issues.apache.org/jira/browse/HBASE-27345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxiang Sun resolved HBASE-27345. -- Fix Version/s: 3.0.0-alpha-4 Assignee: Huaxiang Sun Resolution: Fixed > Add 2.4.14 to the downloads page > > > Key: HBASE-27345 > URL: https://issues.apache.org/jira/browse/HBASE-27345 > Project: HBase > Issue Type: Task > Components: documentation >Affects Versions: 2.4.14 >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Minor > Fix For: 3.0.0-alpha-4 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27345) Add 2.4.14 to the downloads page
Huaxiang Sun created HBASE-27345: Summary: Add 2.4.14 to the downloads page Key: HBASE-27345 URL: https://issues.apache.org/jira/browse/HBASE-27345 Project: HBase Issue Type: Task Components: documentation Affects Versions: 2.4.14 Reporter: Huaxiang Sun -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27152) Under compaction mark may leak
[
https://issues.apache.org/jira/browse/HBASE-27152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaolin Ha resolved HBASE-27152.
Resolution: Fixed
Addressed the UT issues. Merged to branch-2, branch-2.5 and master. Thanks
[~zhangduo] for reviewing.
> Under compaction mark may leak
> --
>
> Key: HBASE-27152
> URL: https://issues.apache.org/jira/browse/HBASE-27152
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Affects Versions: 3.0.0-alpha-2, 2.4.12
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Fix For: 2.6.0, 2.5.1, 3.0.0-alpha-4
>
>
> HBASE-26249 introduced an under compaction mark to reduce repeatedly
> compactions for the same stores for a short period of time after bulk loading
> files.
> Since the mark adding and removing are in difference threads,
> {code:java}
> pool.execute(
> new CompactionRunner(store, region, compaction, tracker, completeTracker,
> pool, user));
> if (LOG.isDebugEnabled()) {
> LOG.debug(
> "Add compact mark for store {}, priority={}, current under compaction "
> + "store size is {}",
> getStoreNameForUnderCompaction(store), priority,
> underCompactionStores.size());
> }
> underCompactionStores.add(getStoreNameForUnderCompaction(store)); {code}
> it happens that the concurrently of thread-1 using
> underCompactionStores.add() and thread-2 which running a CompactionRunner
> that using underCompactionStores.remove(). If the removing happens before
> adding, the mark will leak.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [DISCUSS] HBase 2.5 / Hadoop 3 artifacts
I think changing the default hadoop profile for builds in branch-2 would unnecessarily complicate our compatibility messaging so long as Hadoop 2 hasn't gone EOL. On Mon, Aug 29, 2022 at 5:30 AM Nick Dimiduk wrote: > Should we also make hadoop3 the default active profile for branch-2 going > forward? > > On Fri, Aug 26, 2022 at 5:25 PM Andrew Purtell > wrote: > > > The security posture of Hadoop 2 in general is a problem, because > > maintenance on that branch is spotty, that is just how it goes. We had > the > > same situation with our now EOL branch-1. I know Hadoop released 2.10.2 > to > > address some CVE worthy problems but it is unclear if 2.10.2 addresses > all > > known issues, unlike 3.3.4. Also as you know Hadoop 2 has unpatchable > > dependencies on org.codehaus versions of Jackson and Jetty, which > > themselves have high scoring CVEs that will never be fixed because they > are > > EOL, and other similar issues. Hadoop 3 doesn’t completely solve such > > problems but is the only realistic place we can hope they can be > addressed > > as required. For organizations that implement or require a top to bottom > > security audit of their software bill of materials, it seems possible to > > avoid user pain by providing supported convenience artifacts *and* > > libraries built against Hadoop 3 APIs in the Apache repository > addressable > > with a Maven classifier. > > > > My employer has some interests in this area that align so I would like to > > sponsor (implement, review, commit, RM backfill releases, etc.) this > work. > > Would there be any objections? Read through the thread for some thoughts > on > > approach. Summarized: > > > > - Amend create-release to build, stage, and deploy a -hadoop3 variant > > build by activating the Hadoop 3 build profile. > > > > - Amend the Hadoop 3 build profile to flatten POMs before deployment to > > resolve potential downstream issues due to Hadoop 3 being a non-default > > build profile. (This could also be applied to all builds.) > > > > - Amend hbase-vote to be aware of and evaluate if present -hadoop3 > variant > > artifacts. > > > > > > > On Aug 25, 2022, at 10:40 AM, Andrew Purtell > > > wrote: > > > > > > Thanks, that would work. > > > > > >> On Aug 25, 2022, at 11:35 AM, Sean Busbey wrote: > > >> > > >> yes, the flatten plugin. We use it in hbase-connectors already. > > >> > > >> https://www.mojohaus.org/flatten-maven-plugin/ > > >> > > >> this sounds like it could also be a use case for BOMs, which would > also > > >> benefit users of our client artifacts that use build tools that don't > > >> respect maven profiles generally, like gradle. > > >> > > >>> On Thu, Aug 25, 2022 at 10:30 AM Andrew Purtell < > > andrew.purt...@gmail.com> > > >>> wrote: > > >>> > > >>> Thinking about this a bit more, we will have an issue in that the > POMs > > >>> published from our -hadoop3 build will not have a default activation > > of our > > >>> Hadoop 3 build profile. The convenience binaries will function as > > expected > > >>> but Maven will read and process eg Phoenix POMs, then download and > > perform > > >>> substitutions on HBase POMs, and then etc, so downstreamers like > > Phoenix > > >>> will have to set up the hadoop.profile variable for us in their > default > > >>> build profile or else the transitive paths through us may be wrong. I > > >>> wonder if there is a Maven plugin available for deploying POMs with > all > > >>> variable substitutions performed before deployment, that would solve > > that > > >>> problem and all conceivable related issues. > > >>> > > On Aug 25, 2022, at 11:03 AM, Andrew Purtell < > > andrew.purt...@gmail.com> > > >>> wrote: > > > > I think 2.x is going to have a few years of life remaining so it > > would > > >>> be best, if we are going to address this, to have a 2.x solution was > > well > > >>> as a 3.x one. > > > > In my opinion we can continue to publish 2.4 and 2.5 (and 2.6) > > unchanged > > >>> and then also introduce a Hadoop 3 release using “hadoop3” or similar > > as > > >>> Maven classifier. Phoenix could specify this classifier in their > POMs. > > >>> Everyone should be happy. Users who already are comfortable with the > > Hadoop > > >>> 2 default don’t have to change anything. A one time POM change on the > > >>> Phoenix side is required but that’s it. > > > > The additional build time complexity for generating two releases can > > be > > >>> incorporated into create-release. Nobody does manual releases any > more > > as > > >>> far as I know. Likewise, download and verification of -hadoop3 > > convenience > > >>> binaries can be added to hbase-vote. I believe we are all using that > > tool > > >>> for verification of releases now. After these one time changes are > > landed > > >>> the cost for RMs and PMC will be only in a roughly doubled amount of > > time > > >>> needed to build and verify releases. > > > > >> On Aug 17, 2022, at 9:06 AM, Nick Dimiduk >
[jira] [Resolved] (HBASE-24457) release scripts on mac os are too slow
[ https://issues.apache.org/jira/browse/HBASE-24457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-24457. - Fix Version/s: 3.0.0-alpha-1 (was: 3.0.0-alpha-4) Resolution: Fixed > release scripts on mac os are too slow > -- > > Key: HBASE-24457 > URL: https://issues.apache.org/jira/browse/HBASE-24457 > Project: HBase > Issue Type: Umbrella > Components: community >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 3.0.0-alpha-1 > > > related changes to try to bring the time down to something I can use -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27344) Update ref guide about the upcoming 2.5.0 release
Duo Zhang created HBASE-27344: - Summary: Update ref guide about the upcoming 2.5.0 release Key: HBASE-27344 URL: https://issues.apache.org/jira/browse/HBASE-27344 Project: HBase Issue Type: Task Reporter: Duo Zhang The hadoop support matrix, release manager, etc. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Blog Posts on "How we solved for Hbase MultiTenancy"
Multi Tenancy was needed because of diverse use cases at our org. So we took hbase and made customisations on top of 2.1 to have better isolation guarantees. I have put down those customisations in the following II part articles. Part I --> https://blog.flipkart.tech/hbase-multi-tenancy-part-i-37cad340c0fa Part II --> https://blog.flipkart.tech/hbase-multi-tenancy-part-ii-79488c19b03d Talk: https://www.youtube.com/watch?v=ttGI9Ma7Xos=26s_channel=LifeAtFlipkart Some of those can be contributed back upstream with modifications. Happy to hear thoughts. --- Mallikarjun
Re: How to store keystore/truststore password securely?
Good questions. Looks the API hasn't changed since when it was added in 2014: https://github.com/Jerry-Xin/hadoop/commit/c79728478caadd8374bce2bc3f466db1da1e3ad1 I'll create a patch to support this, apart from the CLI extension this looks like super easy to integrate. Filed jira: https://issues.apache.org/jira/browse/HBASE-27342 Andor On Thu, 2022-08-25 at 20:15 +0200, Nick Dimiduk wrote: > Is the Hadoop Credentials API protected by their compatibility > guidelines? > If not, can we convince them to support it formally? > > On Thu, Aug 25, 2022 at 17:15 Andrew Purtell < > andrew.purt...@gmail.com> > wrote: > > > We can import the code from our sister Apache project if long term > > dependency and compatibility are concerns. The concerns apply in > > both > > directions: > > > > Depend, do not import: Hadoop may break us with a change that we > > have to > > incorporate by updating to a new minimum version probably because > > of a > > security issue. How likely is this? I don’t think the SPI has > > changed much. > > Cannot do a decent review at this time, on phone only. > > > > Import: If we do not watch the Hadoop implementation, then our > > semantics > > may diverge from theirs in a way that is confusing and inconvenient > > for > > operators who have to manage credentials at both Hadoop and HBase > > layers. > > > > > On Aug 25, 2022, at 11:00 AM, 张铎 wrote: > > > > > > +1 > > > > > > But I'm still not sure whether we should just use the code in > > > hadoop, > > > or we should just use the mechanism but write(copy) the code by > > > our > > > own? > > > > > > Andrew Purtell 于2022年8月25日周四 22:13写道： > > > > I agree the Credential SPI provided by Hadoop is direct and > > > > expedient. > > > > > > > > I would ask that a patch integrating it, if this is the > > > > selected > > approach, should also add support to bin/hbase so “hbase credential > > …” > > command line support is available and identical to that provided by > > the > > Hadoop bin script. This is for convenience and also a concession to > > users > > that ship HBase binaries/packages disaggregated from Hadoop ones. > > > > > > On Aug 25, 2022, at 9:50 AM, Andor Molnar > > > > > > wrote: > > > > > > > > > > As Bryan mentioned there's a nice, extensible API already > > > > > available in > > > > > Hadoop, the Credentials API. > > > > > > > > > > > > https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html > > > > > I think that would be the quickest and easiest approach to > > > > > resolve this > > > > > problem. Is there any objection or downside of that? > > > > > > > > > > Andor > > > > > > > > > > > > > > > > > > > > > On Tue, 2022-08-23 at 21:39 +0800, 张铎(Duo Zhang) wrote: > > > > > > Maybe we could introduce two configs for a password, > > > > > > password- > > > > > > provider > > > > > > and password-provider-parameter > > > > > > > > > > > > The default implementation is FileBasedPasswordProvider, > > > > > > where the > > > > > > parameter is just a location. And also an > > > > > > EnvVarPasswordProvider, > > > > > > where the parameter is the name of the environment > > > > > > variable. > > > > > > > > > > > > And users could also implement their own providers. > > > > > > > > > > > > WDYT? > > > > > > > > > > > > Bryan Beaudreault > > > > > > 于2022年8月23日周二 > > > > > > 21:12写道： > > > > > > > I agree that it seems insecure to put it directly into > > > > > > > the hbase- > > > > > > > site.xml. > > > > > > > Another reason is due to the RS UI which (helpfully) can > > > > > > > print the > > > > > > > entire > > > > > > > site configuration. We’d need to make sure the password > > > > > > > is excluded > > > > > > > from > > > > > > > that, but better to remove it from site xml altogether. > > > > > > > > > > > > > > That said, we already have the concept of keystore > > > > > > > passwords in > > > > > > > hbase -- > > > > > > > search our refguide for "password" and you'll see two > > > > > > > examples: for > > > > > > > jmx ssl > > > > > > > and for encryption-at-rest. Both cases seem to take the > > > > > > > approach > > > > > > > of > > > > > > > allowing an explicit password or a password file. Another > > > > > > > example > > > > > > > we can > > > > > > > take inspiration from is Hadoop Credentials API[1] which > > > > > > > allows > > > > > > > specifying > > > > > > > by environment variable or password file. Searching > > > > > > > around for > > > > > > > other > > > > > > > opensource projects, these options seem to be the most > > > > > > > common for > > > > > > > the > > > > > > > keystore password. See Cassandra[2] and Zookeeper[3] as > > > > > > > further > > > > > > > examples. > > > > > > > > > > > > > > Elastic takes the approach of allowing "secure settings" > > > > > > > [4], which > > > > > > > are > > > > > > > stored in a separate keystore managed via elasticsearch- > > > > > > > keystore > > > > > > > command. > > > > > > >
Re: [VOTE] Release candidate for HBase 2.5.0 (RC1) is available
+1 * API Compatibility Report: ok * Nightly CI: ok * * Signature: ok * Checksum : ok * Rat check (1.8.0_292): ok - mvn clean apache-rat:check -D skipTests * Rat check (11.0.11): ok - mvn clean apache-rat:check -D skipTests -D hadoop.profile=3.0 * Built from source (1.8.0_292): ok - mvn clean install -D skipTests -DskipTests * Built from source (11.0.11): ok - mvn clean install -D skipTests -D hadoop.profile=3.0 -DskipTests * There's a couple flakeys in the build history. They almost all fail with timeout. TestSplitRegionWhileRSCrash, TestMergeTableRegionsWhileRSCrash, TestLogRolling, TestFastFail, TestQuotaAdmin, TestUnknownServers, TestAdminShell2 So far, we've had four commits land since the RC. Of this, I'm surprised HBASE-27320 is filed as an "improvement" and not something like "Critical". Perhaps we can have some further discussion on that ticket. Thanks, Nick On Wed, Aug 24, 2022 at 11:52 AM Nick Dimiduk wrote: > Please vote on this Apache hbase release candidate, > hbase-2.5.0RC1 > > The VOTE will remain open for at least 72 hours. > > [ ] +1 Release this package as Apache hbase 2.5.0 > [ ] -1 Do not release this package because ... > > The tag to be voted on is 2.5.0RC1: > > https://github.com/apache/hbase/tree/2.5.0RC1 > > This tag currently points to git reference > > 2ecd8bd6d615ca49bfb329b3c0c126c80846d4ab > > The release files, including signatures, digests, as well as CHANGES.md > and RELEASENOTES.md included in this RC can be found at: > > https://dist.apache.org/repos/dist/dev/hbase/2.5.0RC1/ > > Maven artifacts are available in a staging repository at: > > https://repository.apache.org/content/repositories/orgapachehbase-1494/ > > Artifacts were signed with the 0x18567F39 key which can be found in: > > https://downloads.apache.org/hbase/KEYS > > To learn more about Apache hbase, please see > > http://hbase.apache.org/ > > Thanks, > Your HBase Release Manager >
Re: [DISCUSS] Should we exclude shaded artifacts from the API compatibility report?
Or perhaps we shouldn't skip over these shaded results. I don't actually see these issues reported twice. Please disregard. On Mon, Aug 29, 2022 at 2:19 PM Nick Dimiduk wrote: > Heya, > > Browsing through the report on 2.5.0RC1, I find I'm scrolling past the > changes reported in the shaded artifacts because they are a repeat of > what's observed in the core jars. Because they are generated from the core > libraries, should we exclude these shaded artifacts from the report? Asked > a different way, is there anything specific that a reviewer should be > looking for in the report as pertaining to the shaded artifacts? > > Thanks, > Nick >
[jira] [Created] (HBASE-27343) RoundRobinTableInputFormat main should extend `AbstractHBaseTool` for proper configuration parsing
Nick Dimiduk created HBASE-27343: Summary: RoundRobinTableInputFormat main should extend `AbstractHBaseTool` for proper configuration parsing Key: HBASE-27343 URL: https://issues.apache.org/jira/browse/HBASE-27343 Project: HBase Issue Type: Task Components: mapreduce Affects Versions: 2.4.2, 2.3.5, 3.0.0-alpha-1, 2.5.0 Reporter: Nick Dimiduk Noticed while reviewing the compatibility report for 2.5.0RC1. `RoundRobinTableInputFormat` ships with an undocumented main method that prints out input split diagnostic information. That main does some basic configuration parsing but it inconsistent with how parsing is normally done. If the tool is going to ship a main, it should do paring properly, make use of `AbstractHBaseTool`. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27342) Use Hadoop Credentials API to retrieve passwords of TLS key/trust stores
Andor Molnar created HBASE-27342: Summary: Use Hadoop Credentials API to retrieve passwords of TLS key/trust stores Key: HBASE-27342 URL: https://issues.apache.org/jira/browse/HBASE-27342 Project: HBase Issue Type: Improvement Components: IPC/RPC, security Reporter: Andor Molnar Assignee: Andor Molnar Based on a discussion in the TLS Jira and mailing list, it would be beneficial to protect the password of trust and key stores for TLS encryption support in Netty RPC. [https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[DISCUSS] Should we exclude shaded artifacts from the API compatibility report?
Heya, Browsing through the report on 2.5.0RC1, I find I'm scrolling past the changes reported in the shaded artifacts because they are a repeat of what's observed in the core jars. Because they are generated from the core libraries, should we exclude these shaded artifacts from the report? Asked a different way, is there anything specific that a reviewer should be looking for in the report as pertaining to the shaded artifacts? Thanks, Nick
Re: [DISCUSS] HBase 2.5 / Hadoop 3 artifacts
Should we also make hadoop3 the default active profile for branch-2 going forward? On Fri, Aug 26, 2022 at 5:25 PM Andrew Purtell wrote: > The security posture of Hadoop 2 in general is a problem, because > maintenance on that branch is spotty, that is just how it goes. We had the > same situation with our now EOL branch-1. I know Hadoop released 2.10.2 to > address some CVE worthy problems but it is unclear if 2.10.2 addresses all > known issues, unlike 3.3.4. Also as you know Hadoop 2 has unpatchable > dependencies on org.codehaus versions of Jackson and Jetty, which > themselves have high scoring CVEs that will never be fixed because they are > EOL, and other similar issues. Hadoop 3 doesn’t completely solve such > problems but is the only realistic place we can hope they can be addressed > as required. For organizations that implement or require a top to bottom > security audit of their software bill of materials, it seems possible to > avoid user pain by providing supported convenience artifacts *and* > libraries built against Hadoop 3 APIs in the Apache repository addressable > with a Maven classifier. > > My employer has some interests in this area that align so I would like to > sponsor (implement, review, commit, RM backfill releases, etc.) this work. > Would there be any objections? Read through the thread for some thoughts on > approach. Summarized: > > - Amend create-release to build, stage, and deploy a -hadoop3 variant > build by activating the Hadoop 3 build profile. > > - Amend the Hadoop 3 build profile to flatten POMs before deployment to > resolve potential downstream issues due to Hadoop 3 being a non-default > build profile. (This could also be applied to all builds.) > > - Amend hbase-vote to be aware of and evaluate if present -hadoop3 variant > artifacts. > > > > On Aug 25, 2022, at 10:40 AM, Andrew Purtell > wrote: > > > > Thanks, that would work. > > > >> On Aug 25, 2022, at 11:35 AM, Sean Busbey wrote: > >> > >> yes, the flatten plugin. We use it in hbase-connectors already. > >> > >> https://www.mojohaus.org/flatten-maven-plugin/ > >> > >> this sounds like it could also be a use case for BOMs, which would also > >> benefit users of our client artifacts that use build tools that don't > >> respect maven profiles generally, like gradle. > >> > >>> On Thu, Aug 25, 2022 at 10:30 AM Andrew Purtell < > andrew.purt...@gmail.com> > >>> wrote: > >>> > >>> Thinking about this a bit more, we will have an issue in that the POMs > >>> published from our -hadoop3 build will not have a default activation > of our > >>> Hadoop 3 build profile. The convenience binaries will function as > expected > >>> but Maven will read and process eg Phoenix POMs, then download and > perform > >>> substitutions on HBase POMs, and then etc, so downstreamers like > Phoenix > >>> will have to set up the hadoop.profile variable for us in their default > >>> build profile or else the transitive paths through us may be wrong. I > >>> wonder if there is a Maven plugin available for deploying POMs with all > >>> variable substitutions performed before deployment, that would solve > that > >>> problem and all conceivable related issues. > >>> > On Aug 25, 2022, at 11:03 AM, Andrew Purtell < > andrew.purt...@gmail.com> > >>> wrote: > > I think 2.x is going to have a few years of life remaining so it > would > >>> be best, if we are going to address this, to have a 2.x solution was > well > >>> as a 3.x one. > > In my opinion we can continue to publish 2.4 and 2.5 (and 2.6) > unchanged > >>> and then also introduce a Hadoop 3 release using “hadoop3” or similar > as > >>> Maven classifier. Phoenix could specify this classifier in their POMs. > >>> Everyone should be happy. Users who already are comfortable with the > Hadoop > >>> 2 default don’t have to change anything. A one time POM change on the > >>> Phoenix side is required but that’s it. > > The additional build time complexity for generating two releases can > be > >>> incorporated into create-release. Nobody does manual releases any more > as > >>> far as I know. Likewise, download and verification of -hadoop3 > convenience > >>> binaries can be added to hbase-vote. I believe we are all using that > tool > >>> for verification of releases now. After these one time changes are > landed > >>> the cost for RMs and PMC will be only in a roughly doubled amount of > time > >>> needed to build and verify releases. > > >> On Aug 17, 2022, at 9:06 AM, Nick Dimiduk > wrote: > >> > >> Hi Geoffrey, > >> > >> I have no complaints with shipping convenience binaries built > against > >>> both > > Hadoop2 and Hadoop3. The primary challenge is implementing the > > necessary build changes, the secondary challenge is > verifying/testing it > > works reliably. > > > > But for Phoenix, are you asking for convenience binaries, or are you > >>> asking > > for artifacts published into maven
