> > Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it > mentioned anywhere in change log or release notes. Are they actually > up-to-date?
I don't think there is any issue with the ReleaseNotes generation as such but with the Resolution type of this ticket, It ain't marked as Fixed but Done. The other ticket which is marked Done is also not part of the release notes. [1] if I'm understanding the potential impact of HDFS-16853 > correctly, then it's serious enough to fix before a release. (I could > change my vote if someone wants to make a case that it's not that serious.) > Chris, I just had a very quick look at HDFS-16853, I am not sure if this can happen outside a MiniDfsCluster setup? Just guessing from the description in the ticket. It looked like when we did a restart of the Namenode in the MiniDfsCluster, I guess that would be in the same single JVM, and that is why a previous blocked thread caused issues with the restart. That is what I understood, I haven't checked the code though. Second, In the same context, Being curious If this lands up being a MiniDfsCluster only issue, do we still consider this a release blocker? Not saying in a way it won't be serious, MiniDfsCluster is very widely used by downstream projects and all, so just wanted to know.... Regarding the Hive & Bouncy castle. The PR seems to have a valid binding veto, I am not sure if it will get done any time soon, so if the use case is something required, I would suggest handling it at Hadoop itself. It seems to be centric to Hive-3.x, I tried compiling the Hive master branch with 3.3.5 and it passed. Other than that Hive officially support only Hadoop-3.3.1 and that too only in the last 4.x release[2] [1] https://issues.apache.org/jira/browse/HADOOP-11867?jql=project%20%3D%20HADOOP%20AND%20resolution%20%3D%20Done%20AND%20fixVersion%20%3D%203.3.5%20ORDER%20BY%20resolution%20DESC [2] https://issues.apache.org/jira/browse/HIVE-24484 -Ayush On Tue, 3 Jan 2023 at 23:51, Chris Nauroth <cnaur...@apache.org> wrote: > -1, because if I'm understanding the potential impact of HDFS-16853 > correctly, then it's serious enough to fix before a release. (I could > change my vote if someone wants to make a case that it's not that serious.) > > Otherwise, this RC was looking good: > > * Verified all checksums. > * Verified all signatures. > * Built from source, including native code on Linux. > * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy > -Drequire.zstd -DskipTests > * Tests passed. > * mvn --fail-never clean test -Pnative -Dparallel-tests > -Drequire.snappy -Drequire.zstd -Drequire.openssl > -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8 > * Checked dependency tree to make sure we have all of the expected library > updates that are mentioned in the release notes. > * mvn -o dependency:tree > * Farewell, S3Guard. > * Confirmed that hadoop-openstack is now just a stub placeholder artifact > with no code. > * For ARM verification: > * Ran "file <X>" on all native binaries in the ARM tarball to confirm > they actually came out with ARM as the architecture. > * Output of hadoop checknative -a on ARM looks good. > * Ran a MapReduce job with the native bzip2 codec for compression, and > it worked fine. > * Ran a MapReduce job with YARN configured to use > LinuxContainerExecutor and verified launching the containers through > container-executor worked. > > My local setup didn't have the test failures mentioned by Viraj, though > there was some flakiness with a few HDFS snapshot tests timing out. > > Regarding Hive and Bouncy Castle, there is an existing issue and pull > request tracking an upgrade attempt. It's looking like some amount of code > changes are required: > > https://issues.apache.org/jira/browse/HIVE-26648 > https://github.com/apache/hive/pull/3744 > > Chris Nauroth > > > On Tue, Jan 3, 2023 at 8:57 AM Chao Sun <sunc...@apache.org> wrote: > > > Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it > > mentioned anywhere in change log or release notes. Are they actually > > up-to-date? > > > > On Mon, Jan 2, 2023 at 7:48 AM Masatake Iwasaki > > <iwasak...@oss.nttdata.com> wrote: > > > > > > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to > > dependency change. > > > > > > For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are > not > > expected in hbase-shaded-with-hadoop-check-invariants. > > > Updating hbase-shaded/pom.xml is expected to be the fix as done in > > HBASE-27292. > > > > > > https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277 > > > > > > [INFO] --- exec-maven-plugin:1.6.0:exec > > (check-jar-contents-for-stuff-with-hadoop) @ > > hbase-shaded-with-hadoop-check-invariants --- > > > [ERROR] Found artifact with unexpected contents: > > > '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar' > > > Please check the following and either correct the build or > update > > > the allowed list with reasoning. > > > > > > com/ > > > com/sun/ > > > com/sun/jersey/ > > > com/sun/jersey/json/ > > > ... > > > > > > > > > For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68 > seem > > to be problematic. > > > Excluding them on hive-jdbc might be the fix. > > > > > > [ERROR] Failed to execute goal > > org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on > > project hive-jdbc: Error creating shaded jar: Problem shading JAR > > > /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar > > entry > > > META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class: > > java.lang.IllegalArgumentException: Unsupported class file major version > 59 > > -> [Help 1] > > > ... > > > > > > > > > On 2023/01/02 22:02, Masatake Iwasaki wrote: > > > > Thanks for your great effort for the new release, Steve and Mukund. > > > > > > > > +1 while it would be nice if we can address missed Javadocs. > > > > > > > > + verified the signature and checksum. > > > > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with > native > > profile enabled. > > > > + launched pseudo distributed cluster including kms and httpfs > with > > Kerberos and SSL enabled. > > > > + created encryption zone, put and read files via httpfs. > > > > + ran example MR wordcount over encryption zone. > > > > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8 > > (both x86_64 and aarch64). > > > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to > > dependency change. > > > > # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop > 3.3.4 > > worked. > > > > + skimmed the site contents. > > > > - Javadocs are not contained (under r3.3.5/api). > > > > # The issue can be reproduced even if I built site docs from the > > source. > > > > > > > > Masatake Iwasaki > > > > > > > > On 2022/12/22 4:28, Steve Loughran wrote: > > > >> Mukund and I have put together a release candidate (RC0) for Hadoop > > 3.3.5. > > > >> > > > >> Given the time of year it's a bit unrealistic to run a 5 day vote > and > > > >> expect people to be able to test it thoroughly enough to make this > > the one > > > >> we can ship. > > > >> > > > >> What we would like is for anyone who can to verify the tarballs, and > > test > > > >> the binaries, especially anyone who can try the arm64 binaries. > We've > > got > > > >> the building of those done and now the build file will incorporate > > them > > > >> into the release -but neither of us have actually tested it yet. > > Maybe I > > > >> should try it on my pi400 over xmas. > > > >> > > > >> The maven artifacts are up on the apache staging repo -they are the > > ones > > > >> from x86 build. Building and testing downstream apps will be > > incredibly > > > >> helpful. > > > >> > > > >> The RC is available at: > > > >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/ > > > >> > > > >> The git tag is release-3.3.5-RC0, commit 3262495904d > > > >> > > > >> The maven artifacts are staged at > > > >> > > https://repository.apache.org/content/repositories/orgapachehadoop-1365/ > > > >> > > > >> You can find my public key at: > > > >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > >> > > > >> Change log > > > >> > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md > > > >> > > > >> Release notes > > > >> > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md > > > >> > > > >> This is off branch-3.3 and is the first big release since 3.3.2. > > > >> > > > >> Key changes include > > > >> > > > >> * Big update of dependencies to try and keep those reports of > > > >> transitive CVEs under control -both genuine and false positive. > > > >> * HDFS RBF enhancements > > > >> * Critical fix to ABFS input stream prefetching for correct reading. > > > >> * Vectored IO API for all FSDataInputStream implementations, with > > > >> high-performance versions for file:// and s3a:// filesystems. > > > >> file:// through java native io > > > >> s3a:// parallel GET requests. > > > >> * This release includes Arm64 binaries. Please can anyone with > > > >> compatible systems validate these. > > > >> > > > >> > > > >> Please try the release and vote on it, even though i don't know what > > is a > > > >> good timeline here...i'm actually going on holiday in early jan. > > Mukund is > > > >> around and so can drive the process while I'm offline. > > > >> > > > >> Assuming we do have another iteration, the RC1 will not be before > mid > > jan > > > >> for that reason > > > >> > > > >> Steve (and mukund) > > > >> > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > > > > For additional commands, e-mail: common-dev-h...@hadoop.apache.org > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > > For additional commands, e-mail: common-dev-h...@hadoop.apache.org > > > > >
