Is it a problem limited to MiniDFSCluster, or is it a broader problem of RPC client resource cleanup? The patch is changing connection close cleanup, so I assumed the latter. If so, then it could potentially impact applications integrating with the RPC clients.
If the problem is limited to MiniDFSCluster and restarts within a single JVM, then I agree the impact is smaller. Then, we'd want to consider what downstream projects have tests that do restarts on a MiniDFSCluster. Chris Nauroth On Wed, Jan 4, 2023 at 4:22 PM Ayush Saxena <ayush...@gmail.com> wrote: > Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it >> mentioned anywhere in change log or release notes. Are they actually >> up-to-date? > > > I don't think there is any issue with the ReleaseNotes generation as such > but with the Resolution type of this ticket, It ain't marked as Fixed but > Done. The other ticket which is marked Done is also not part of the release > notes. [1] > > if I'm understanding the potential impact of HDFS-16853 >> correctly, then it's serious enough to fix before a release. (I could >> change my vote if someone wants to make a case that it's not that >> serious.) >> > > Chris, I just had a very quick look at HDFS-16853, I am not sure if this > can happen outside a MiniDfsCluster setup? Just guessing from the > description in the ticket. It looked like when we did a restart of the > Namenode in the MiniDfsCluster, I guess that would be in the same single > JVM, and that is why a previous blocked thread caused issues with the > restart. That is what I understood, I haven't checked the code though. > > Second, In the same context, Being curious If this lands up being a > MiniDfsCluster only issue, do we still consider this a release blocker? Not > saying in a way it won't be serious, MiniDfsCluster is very widely used by > downstream projects and all, so just wanted to know.... > > Regarding the Hive & Bouncy castle. The PR seems to have a valid binding > veto, I am not sure if it will get done any time soon, so if the use case > is something required, I would suggest handling it at Hadoop itself. It > seems to be centric to Hive-3.x, I tried compiling the Hive master branch > with 3.3.5 and it passed. Other than that Hive officially support only > Hadoop-3.3.1 and that too only in the last 4.x release[2] > > > [1] > https://issues.apache.org/jira/browse/HADOOP-11867?jql=project%20%3D%20HADOOP%20AND%20resolution%20%3D%20Done%20AND%20fixVersion%20%3D%203.3.5%20ORDER%20BY%20resolution%20DESC > [2] https://issues.apache.org/jira/browse/HIVE-24484 > > -Ayush > > On Tue, 3 Jan 2023 at 23:51, Chris Nauroth <cnaur...@apache.org> wrote: > >> -1, because if I'm understanding the potential impact of HDFS-16853 >> correctly, then it's serious enough to fix before a release. (I could >> change my vote if someone wants to make a case that it's not that >> serious.) >> >> Otherwise, this RC was looking good: >> >> * Verified all checksums. >> * Verified all signatures. >> * Built from source, including native code on Linux. >> * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy >> -Drequire.zstd -DskipTests >> * Tests passed. >> * mvn --fail-never clean test -Pnative -Dparallel-tests >> -Drequire.snappy -Drequire.zstd -Drequire.openssl >> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8 >> * Checked dependency tree to make sure we have all of the expected library >> updates that are mentioned in the release notes. >> * mvn -o dependency:tree >> * Farewell, S3Guard. >> * Confirmed that hadoop-openstack is now just a stub placeholder artifact >> with no code. >> * For ARM verification: >> * Ran "file <X>" on all native binaries in the ARM tarball to confirm >> they actually came out with ARM as the architecture. >> * Output of hadoop checknative -a on ARM looks good. >> * Ran a MapReduce job with the native bzip2 codec for compression, and >> it worked fine. >> * Ran a MapReduce job with YARN configured to use >> LinuxContainerExecutor and verified launching the containers through >> container-executor worked. >> >> My local setup didn't have the test failures mentioned by Viraj, though >> there was some flakiness with a few HDFS snapshot tests timing out. >> >> Regarding Hive and Bouncy Castle, there is an existing issue and pull >> request tracking an upgrade attempt. It's looking like some amount of code >> changes are required: >> >> https://issues.apache.org/jira/browse/HIVE-26648 >> https://github.com/apache/hive/pull/3744 >> >> Chris Nauroth >> >> >> On Tue, Jan 3, 2023 at 8:57 AM Chao Sun <sunc...@apache.org> wrote: >> >> > Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it >> > mentioned anywhere in change log or release notes. Are they actually >> > up-to-date? >> > >> > On Mon, Jan 2, 2023 at 7:48 AM Masatake Iwasaki >> > <iwasak...@oss.nttdata.com> wrote: >> > > >> > > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due >> to >> > dependency change. >> > > >> > > For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are >> not >> > expected in hbase-shaded-with-hadoop-check-invariants. >> > > Updating hbase-shaded/pom.xml is expected to be the fix as done in >> > HBASE-27292. >> > > >> > >> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277 >> > > >> > > [INFO] --- exec-maven-plugin:1.6.0:exec >> > (check-jar-contents-for-stuff-with-hadoop) @ >> > hbase-shaded-with-hadoop-check-invariants --- >> > > [ERROR] Found artifact with unexpected contents: >> > >> '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar' >> > > Please check the following and either correct the build or >> update >> > > the allowed list with reasoning. >> > > >> > > com/ >> > > com/sun/ >> > > com/sun/jersey/ >> > > com/sun/jersey/json/ >> > > ... >> > > >> > > >> > > For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68 >> seem >> > to be problematic. >> > > Excluding them on hive-jdbc might be the fix. >> > > >> > > [ERROR] Failed to execute goal >> > org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on >> > project hive-jdbc: Error creating shaded jar: Problem shading JAR >> > >> /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar >> > entry >> > >> META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class: >> > java.lang.IllegalArgumentException: Unsupported class file major >> version 59 >> > -> [Help 1] >> > > ... >> > > >> > > >> > > On 2023/01/02 22:02, Masatake Iwasaki wrote: >> > > > Thanks for your great effort for the new release, Steve and Mukund. >> > > > >> > > > +1 while it would be nice if we can address missed Javadocs. >> > > > >> > > > + verified the signature and checksum. >> > > > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with >> native >> > profile enabled. >> > > > + launched pseudo distributed cluster including kms and httpfs >> with >> > Kerberos and SSL enabled. >> > > > + created encryption zone, put and read files via httpfs. >> > > > + ran example MR wordcount over encryption zone. >> > > > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8 >> > (both x86_64 and aarch64). >> > > > - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due >> to >> > dependency change. >> > > > # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop >> 3.3.4 >> > worked. >> > > > + skimmed the site contents. >> > > > - Javadocs are not contained (under r3.3.5/api). >> > > > # The issue can be reproduced even if I built site docs from >> the >> > source. >> > > > >> > > > Masatake Iwasaki >> > > > >> > > > On 2022/12/22 4:28, Steve Loughran wrote: >> > > >> Mukund and I have put together a release candidate (RC0) for Hadoop >> > 3.3.5. >> > > >> >> > > >> Given the time of year it's a bit unrealistic to run a 5 day vote >> and >> > > >> expect people to be able to test it thoroughly enough to make this >> > the one >> > > >> we can ship. >> > > >> >> > > >> What we would like is for anyone who can to verify the tarballs, >> and >> > test >> > > >> the binaries, especially anyone who can try the arm64 binaries. >> We've >> > got >> > > >> the building of those done and now the build file will incorporate >> > them >> > > >> into the release -but neither of us have actually tested it yet. >> > Maybe I >> > > >> should try it on my pi400 over xmas. >> > > >> >> > > >> The maven artifacts are up on the apache staging repo -they are the >> > ones >> > > >> from x86 build. Building and testing downstream apps will be >> > incredibly >> > > >> helpful. >> > > >> >> > > >> The RC is available at: >> > > >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/ >> > > >> >> > > >> The git tag is release-3.3.5-RC0, commit 3262495904d >> > > >> >> > > >> The maven artifacts are staged at >> > > >> >> > >> https://repository.apache.org/content/repositories/orgapachehadoop-1365/ >> > > >> >> > > >> You can find my public key at: >> > > >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS >> > > >> >> > > >> Change log >> > > >> >> > >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md >> > > >> >> > > >> Release notes >> > > >> >> > >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md >> > > >> >> > > >> This is off branch-3.3 and is the first big release since 3.3.2. >> > > >> >> > > >> Key changes include >> > > >> >> > > >> * Big update of dependencies to try and keep those reports of >> > > >> transitive CVEs under control -both genuine and false positive. >> > > >> * HDFS RBF enhancements >> > > >> * Critical fix to ABFS input stream prefetching for correct >> reading. >> > > >> * Vectored IO API for all FSDataInputStream implementations, with >> > > >> high-performance versions for file:// and s3a:// filesystems. >> > > >> file:// through java native io >> > > >> s3a:// parallel GET requests. >> > > >> * This release includes Arm64 binaries. Please can anyone with >> > > >> compatible systems validate these. >> > > >> >> > > >> >> > > >> Please try the release and vote on it, even though i don't know >> what >> > is a >> > > >> good timeline here...i'm actually going on holiday in early jan. >> > Mukund is >> > > >> around and so can drive the process while I'm offline. >> > > >> >> > > >> Assuming we do have another iteration, the RC1 will not be before >> mid >> > jan >> > > >> for that reason >> > > >> >> > > >> Steve (and mukund) >> > > >> >> > > > >> > > > >> --------------------------------------------------------------------- >> > > > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org >> > > > For additional commands, e-mail: common-dev-h...@hadoop.apache.org >> > > > >> > > >> > > --------------------------------------------------------------------- >> > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org >> > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org >> > > >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org >> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org >> > >> > >> >
