CCing common-dev@hadoop.apache.org On Thu, 29 Feb 2024 at 22:36, PJ Fanning <fannin...@apache.org> wrote: > > There is an issue with the protobuf lib - described here [1] > > The idea would be to do a new hadoop-thirdparty release and uptake that. > > Related the hadoop-thirdparty uptake, I would like to get the Avro > uptake merged [2]. I think if we don't merge this for Hadoop 3.4.0, we > will have to wait until v3.5.0 instead because changing the Avro > compilation is probably something that you would want in a patch > release. > > > [1] https://issues.apache.org/jira/browse/HADOOP-19090 > [2] https://github.com/apache/hadoop/pull/4854#issuecomment-1967549235 > > > On Thu, 29 Feb 2024 at 22:24, slfan1989 <slfan1...@apache.org> wrote: > > > > I am preparing hadoop-3.4.0-RC3 as we have already released 3 RC versions > > before, and I hope hadoop-3.4.0-RC3 will receive the approval of the > > members. > > > > Compared to hadoop-3.4.0-RC2, my plan is to backport 2 PRs from branch-3.4 > > to branch-3.4.0: > > > > HADOOP-18088: Replacing log4j 1.x with reload4j. > > HADOOP-19084: Pruning hadoop-common transitive dependencies. > > > > I will use hadoop-release-support to package the arm version. > > > > I plan to release hadoop-3.4.0-RC3 next Monday. > > > > Best Regards, > > Shilun Fan. > > > > On Sat, Feb 24, 2024 at 11:28 AM slfan1989 <slfan1...@apache.org> wrote: > > > > > Thank you very much for Steve's detailed test report and issue > > > description! > > > > > > I appreciate your time spent helping with validation. I am currently > > > trying to use hadoop-release-support to prepare hadoop-3.4.0-RC3. > > > > > > After completing the hadoop-3.4.0 version, I will document some of the > > > issues encountered in the "how to release" document, so that future > > > members > > > can refer to it during the release process. > > > > > > Once again, thank you to all members involved in the hadoop-3.4.0 release. > > > > > > Let's hope for a smooth release process. > > > > > > Best Regards, > > > Shilun Fan. > > > > > > On Sat, Feb 24, 2024 at 2:29 AM Steve Loughran > > > <ste...@cloudera.com.invalid> > > > wrote: > > > > > >> I have been testing this all week, and a -1 until some very minor changes > > >> go in. > > >> > > >> > > >> 1. build the arm64 binaries with the same jar artifacts as the x86 one > > >> 2. include ad8b6541117b HADOOP-18088. Replace log4j 1.x with reload4j. > > >> 3. include 80b4bb68159c HADOOP-19084. Prune hadoop-common transitive > > >> dependencies > > >> > > >> > > >> For #1 we have automation there in my client-validator module, which I > > >> have > > >> moved to be a hadoop-managed project and tried to make more > > >> manageable > > >> https://github.com/apache/hadoop-release-support > > >> > > >> This contains an ant project to perform a lot of the documented build > > >> stages, including using SCP to copy down an x86 release tarball and make > > >> a > > >> signed copy of this containing (locally built) arm artifacts. > > >> > > >> Although that only works with my development environment (macbook m1 > > >> laptop > > >> and remote ec2 server), it should be straightforward to make it more > > >> flexible. > > >> > > >> It also includes and tests a maven project which imports many of the > > >> hadoop-* pom files and run some test with it; this caught some problems > > >> with exported slf4j and log4j2 artifacts getting into the classpath. That > > >> is: hadoop-common pulling in log4j 1.2 and 2.x bindings. > > >> > > >> HADOOP-19084 fixes this; the build file now includes a target to scan the > > >> dependencies and fail if "forbidden" artifacts are found. I have not been > > >> able to stop logback ending on the transitive dependency list, but at > > >> least > > >> there is only one slf4j there. > > >> > > >> HADOOP-18088. Replace log4j 1.x with reload4j switches over to reload4j > > >> while the move to v2 is still something we have to consider a WiP. > > >> > > >> I have tried doing some other changes to the packaging this week > > >> - creating a lean distro without the AWS SDK > > >> - trying to get protobuf-2.5 out of yarn-api > > >> However, I think it is too late to try applying patches this risky. > > >> > > >> I Believe we should get the 3.4.0 release out for people to start playing > > >> with while we rapidly iterate 3.4.1 release out with > > >> - updated dependencies (where possible) > > >> - separate "lean" and "full" installations, where "full" includes all the > > >> cloud connectors and their dependencies; the default is lean and doesn't. > > >> That will cut the default download size in half. > > >> - critical issues which people who use the 3.4.0 release raise with us. > > >> > > >> That is: a packaging and bugs release, with a minimal number of new > > >> features. > > >> > > >> I've created HADOOP-19087 > > >> <https://issues.apache.org/jira/browse/HADOOP-19087> to cover this, > > >> I'm willing to get my hands dirty here -Shilun Fan and Xiaoqiao He have > > >> put > > >> a lot of work on 3.4.0 and probably need other people to take up the work > > >> for next release. Who else is willing to participate? (Yes Mukund, I have > > >> you in mind too) > > >> > > >> One thing I would like to visit is: what hadoop-tools modules can we cut? > > >> Are rumen and hadoop-streaming being actively used? Or can we consider > > >> them > > >> implicitly EOL and strip. Just think of the maintenance effort we would > > >> save. > > >> > > >> --- > > >> > > >> Incidentally, I have tested the arm stuff on my raspberry pi5 which is > > >> now > > >> running 64 bit linux. I believe it is the first time we have qualified a > > >> Hadoop release with the media player under someone's television. > > >> > > >> On Thu, 15 Feb 2024 at 20:41, Mukund Madhav Thakur <mtha...@cloudera.com> > > >> wrote: > > >> > > >> > Thanks, Shilun for putting this together. > > >> > > > >> > Tried the below things and everything worked for me. > > >> > > > >> > validated checksum and gpg signature. > > >> > compiled from source. > > >> > Ran AWS integration tests. > > >> > untar the binaries and able to access objects in S3 via hadoop fs > > >> commands. > > >> > compiled gcs-connector successfully using the 3.4.0 version. > > >> > > > >> > qq: what is the difference between RC1 and RC2? apart from some extra > > >> > patches. > > >> > > > >> > > > >> > > > >> > On Thu, Feb 15, 2024 at 10:58 AM slfan1989 <slfan1...@apache.org> > > >> wrote: > > >> > > > >> >> Thank you for explaining this part! > > >> >> > > >> >> hadoop-3.4.0-RC2 used the validate-hadoop-client-artifacts tool to > > >> >> generate > > >> >> the ARM tar package, which should meet expectations. > > >> >> > > >> >> We also look forward to other members helping to verify. > > >> >> > > >> >> Best Regards, > > >> >> Shilun Fan. > > >> >> > > >> >> On Fri, Feb 16, 2024 at 12:22 AM Steve Loughran <ste...@cloudera.com> > > >> >> wrote: > > >> >> > > >> >> > > > >> >> > > > >> >> > On Mon, 12 Feb 2024 at 15:32, slfan1989 <slfan1...@apache.org> > > >> wrote: > > >> >> > > > >> >> >> > > >> >> >> > > >> >> >> Note, because the arm64 binaries are built separately on a > > >> >> >> different > > >> >> >> platform and JVM, their jar files may not match those of the x86 > > >> >> >> release -and therefore the maven artifacts. I don't think this is > > >> >> >> an issue (the ASF actually releases source tarballs, the binaries > > >> are > > >> >> >> there for help only, though with the maven repo that's a bit > > >> blurred). > > >> >> >> > > >> >> >> The only way to be consistent would actually untar the x86.tar.gz, > > >> >> >> overwrite its binaries with the arm stuff, retar, sign and push out > > >> >> >> for the vote. > > >> >> > > > >> >> > > > >> >> > > > >> >> > that's exactly what the "arm.release" target in my client-validator > > >> >> does. > > >> >> > builds an arm tar with the x86 binaries but the arm native libs, > > >> signs > > >> >> it. > > >> >> > > > >> >> > > > >> >> > > > >> >> >> Even automating that would be risky. > > >> >> >> > > >> >> >> > > >> >> > automating is the *only* way to do it; apache ant has everything > > >> needed > > >> >> > for this including the ability to run gpg. > > >> >> > > > >> >> > we did this on the relevant 3.3.x releases and nobody has yet > > >> >> complained... > > >> >> > > > >> >> > > >> > > > >> > > >
--------------------------------------------------------------------- To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
