-1, because if I'm understanding the potential impact of HDFS-16853
correctly, then it's serious enough to fix before a release. (I could
change my vote if someone wants to make a case that it's not that serious.)
Otherwise, this RC was looking good:
* Verified all checksums.
* Verified all signatures.
* Built from source, including native code on Linux.
* mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
-Drequire.zstd -DskipTests
* Tests passed.
* mvn --fail-never clean test -Pnative -Dparallel-tests
-Drequire.snappy -Drequire.zstd -Drequire.openssl
-Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
* Checked dependency tree to make sure we have all of the expected library
updates that are mentioned in the release notes.
* mvn -o dependency:tree
* Farewell, S3Guard.
* Confirmed that hadoop-openstack is now just a stub placeholder artifact
with no code.
* For ARM verification:
* Ran "file <X>" on all native binaries in the ARM tarball to confirm
they actually came out with ARM as the architecture.
* Output of hadoop checknative -a on ARM looks good.
* Ran a MapReduce job with the native bzip2 codec for compression, and
it worked fine.
* Ran a MapReduce job with YARN configured to use
LinuxContainerExecutor and verified launching the containers through
container-executor worked.
My local setup didn't have the test failures mentioned by Viraj, though
there was some flakiness with a few HDFS snapshot tests timing out.
Regarding Hive and Bouncy Castle, there is an existing issue and pull
request tracking an upgrade attempt. It's looking like some amount of code
changes are required:
https://issues.apache.org/jira/browse/HIVE-26648
https://github.com/apache/hive/pull/3744
Chris Nauroth
On Tue, Jan 3, 2023 at 8:57 AM Chao Sun <sunc...@apache.org> wrote:
> Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> mentioned anywhere in change log or release notes. Are they actually
> up-to-date?
>
> On Mon, Jan 2, 2023 at 7:48 AM Masatake Iwasaki
> <iwasak...@oss.nttdata.com> wrote:
> >
> > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> dependency change.
> >
> > For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are not
> expected in hbase-shaded-with-hadoop-check-invariants.
> > Updating hbase-shaded/pom.xml is expected to be the fix as done in
> HBASE-27292.
> >
> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
> >
> > [INFO] --- exec-maven-plugin:1.6.0:exec
> (check-jar-contents-for-stuff-with-hadoop) @
> hbase-shaded-with-hadoop-check-invariants ---
> > [ERROR] Found artifact with unexpected contents:
> '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
> > Please check the following and either correct the build or update
> > the allowed list with reasoning.
> >
> > com/
> > com/sun/
> > com/sun/jersey/
> > com/sun/jersey/json/
> > ...
> >
> >
> > For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68 seem
> to be problematic.
> > Excluding them on hive-jdbc might be the fix.
> >
> > [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on
> project hive-jdbc: Error creating shaded jar: Problem shading JAR
> /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar
> entry
> META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class:
> java.lang.IllegalArgumentException: Unsupported class file major version 59
> -> [Help 1]
> > ...
> >
> >
> > On 2023/01/02 22:02, Masatake Iwasaki wrote:
> > > Thanks for your great effort for the new release, Steve and Mukund.
> > >
> > > +1 while it would be nice if we can address missed Javadocs.
> > >
> > > + verified the signature and checksum.
> > > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with native
> profile enabled.
> > > + launched pseudo distributed cluster including kms and httpfs with
> Kerberos and SSL enabled.
> > > + created encryption zone, put and read files via httpfs.
> > > + ran example MR wordcount over encryption zone.
> > > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8
> (both x86_64 and aarch64).
> > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> dependency change.
> > > # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop 3.3.4
> worked.
> > > + skimmed the site contents.
> > > - Javadocs are not contained (under r3.3.5/api).
> > > # The issue can be reproduced even if I built site docs from the
> source.
> > >
> > > Masatake Iwasaki
> > >
> > > On 2022/12/22 4:28, Steve Loughran wrote:
> > >> Mukund and I have put together a release candidate (RC0) for Hadoop
> 3.3.5.
> > >>
> > >> Given the time of year it's a bit unrealistic to run a 5 day vote and
> > >> expect people to be able to test it thoroughly enough to make this
> the one
> > >> we can ship.
> > >>
> > >> What we would like is for anyone who can to verify the tarballs, and
> test
> > >> the binaries, especially anyone who can try the arm64 binaries. We've
> got
> > >> the building of those done and now the build file will incorporate
> them
> > >> into the release -but neither of us have actually tested it yet.
> Maybe I
> > >> should try it on my pi400 over xmas.
> > >>
> > >> The maven artifacts are up on the apache staging repo -they are the
> ones
> > >> from x86 build. Building and testing downstream apps will be
> incredibly
> > >> helpful.
> > >>
> > >> The RC is available at:
> > >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
> > >>
> > >> The git tag is release-3.3.5-RC0, commit 3262495904d
> > >>
> > >> The maven artifacts are staged at
> > >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1365/
> > >>
> > >> You can find my public key at:
> > >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >>
> > >> Change log
> > >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
> > >>
> > >> Release notes
> > >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
> > >>
> > >> This is off branch-3.3 and is the first big release since 3.3.2.
> > >>
> > >> Key changes include
> > >>
> > >> * Big update of dependencies to try and keep those reports of
> > >> transitive CVEs under control -both genuine and false positive.
> > >> * HDFS RBF enhancements
> > >> * Critical fix to ABFS input stream prefetching for correct reading.
> > >> * Vectored IO API for all FSDataInputStream implementations, with
> > >> high-performance versions for file:// and s3a:// filesystems.
> > >> file:// through java native io
> > >> s3a:// parallel GET requests.
> > >> * This release includes Arm64 binaries. Please can anyone with
> > >> compatible systems validate these.
> > >>
> > >>
> > >> Please try the release and vote on it, even though i don't know what
> is a
> > >> good timeline here...i'm actually going on holiday in early jan.
> Mukund is
> > >> around and so can drive the process while I'm offline.
> > >>
> > >> Assuming we do have another iteration, the RC1 will not be before mid
> jan
> > >> for that reason
> > >>
> > >> Steve (and mukund)
> > >>
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>
