Hmm interesting. Let me check on this error. Thanks Mukund. Chao
On Fri, Jan 21, 2022 at 4:42 AM Mukund Madhav Thakur <mtha...@cloudera.com.invalid> wrote: > Checked out the release tag. commit *6da346a358c * > Seeing below error while compiling : > > Duplicate classes found: > > > Found in: > > org.apache.hadoop:hadoop-client-api:jar:3.3.2:compile > > org.apache.hadoop:hadoop-client-minicluster:jar:3.3.2:compile > > Duplicate classes: > > org/apache/hadoop/io/serializer/avro/AvroRecord.class > > org/apache/hadoop/io/serializer/avro/AvroRecord$Builder.class > > org/apache/hadoop/io/serializer/avro/AvroRecord$1.class > > > [*INFO*] > *------------------------------------------------------------------------* > > [*INFO*] *Reactor Summary for Apache Hadoop Client Test Minicluster 3.3.2:* > > [*INFO*] > > [*INFO*] Apache Hadoop Client Test Minicluster .............. *SUCCESS* > [02:17 min] > > [*INFO*] Apache Hadoop Client Packaging Invariants for Test . > *FAILURE* [ 0.221 > s] > > [*INFO*] Apache Hadoop Client Packaging Integration Tests ... *SKIPPED* > > [*INFO*] Apache Hadoop Distribution ......................... *SKIPPED* > > [*INFO*] Apache Hadoop Client Modules ....................... *SKIPPED* > > [*INFO*] Apache Hadoop Tencent COS Support .................. *SKIPPED* > > [*INFO*] Apache Hadoop Cloud Storage ........................ *SKIPPED* > > [*INFO*] Apache Hadoop Cloud Storage Project ................ *SKIPPED* > > [*INFO*] > *------------------------------------------------------------------------* > > [*INFO*] *BUILD FAILURE* > > [*INFO*] > *------------------------------------------------------------------------* > > [*INFO*] Total time: 02:18 min > > [*INFO*] Finished at: 2022-01-21T18:06:11+05:30 > > [*INFO*] > *------------------------------------------------------------------------* > > [*ERROR*] Failed to execute goal > org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce > *(enforce-banned-dependencies)* on project > hadoop-client-check-test-invariants: *Some Enforcer rules have failed. Look > above for specific messages explaining why the rule failed.* -> *[Help 1]* > > > On Fri, Jan 21, 2022 at 9:38 AM Wei-Chiu Chuang <weic...@apache.org> > wrote: > > > I'll find time to check out the RC bits. > > I just feel bad that the tarball is now more than 600MB in size. > > > > On Fri, Jan 21, 2022 at 2:23 AM Steve Loughran > <ste...@cloudera.com.invalid > > > > > wrote: > > > > > *+1 binding.* > > > > > > reviewed binaries, source, artifacts in the staging maven repository in > > > downstream builds. all good. > > > > > > *## test run* > > > > > > checked out the asf github repo at commit 6da346a358c into a location > > > already set up with aws and azure test credentials > > > > > > ran the hadoop-aws tests with -Dparallel-tests -DtestsThreadCount=6 > > > -Dmarkers=delete -Dscale > > > and hadoop-azure against azure cardiff with -Dparallel-tests=abfs > > > -DtestsThreadCount=6 > > > > > > all happy > > > > > > > > > > > > *## binary* > > > downloaded KEYS and imported, so adding your key to my list (also > signed > > > this and updated the key servers) > > > > > > downloaded rc tar and verified > > > ``` > > > > gpg2 --verify hadoop-3.3.2.tar.gz.asc hadoop-3.3.2.tar.gz > > > gpg: Signature made Sat Jan 15 23:41:10 2022 GMT > > > gpg: using RSA key > > DE7FA241EB298D027C97B2A1D8F1A97BE51ECA98 > > > gpg: Good signature from "Chao Sun (CODE SIGNING KEY) < > > sunc...@apache.org > > > >" > > > [full] > > > > > > > > > > cat hadoop-3.3.2.tar.gz.sha512 > > > SHA512 (hadoop-3.3.2.tar.gz) = > > > > > > > > > cdd3d9298ba7d6e63ed63f93c159729ea14d2b7d5e3a0640b1761c86c7714a721f88bdfa8cb1d8d3da316f616e4f0ceaace4f32845ee4441e6aaa7a12b8c647d > > > > > > > shasum -a 512 hadoop-3.3.2.tar.gz > > > > > > > > > cdd3d9298ba7d6e63ed63f93c159729ea14d2b7d5e3a0640b1761c86c7714a721f88bdfa8cb1d8d3da316f616e4f0ceaace4f32845ee4441e6aaa7a12b8c647d > > > hadoop-3.3.2.tar.gz > > > ``` > > > > > > > > > *# cloudstore against staged artifacts* > > > ``` > > > cd ~/.m2/repository/org/apache/hadoop > > > find . -name \*3.3.2\* -print | xargs rm -r > > > ``` > > > ensures no local builds have tainted the repo. > > > > > > in cloudstore mvn build without tests > > > ``` > > > mci -Pextra -Phadoop-3.3.2 -Psnapshots-and-staging > > > ``` > > > this fetches all from asf staging > > > > > > ``` > > > Downloading from ASF Staging: > > > > > > > > > https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-client/3.3.2/hadoop-client-3.3.2.pom > > > Downloaded from ASF Staging: > > > > > > > > > https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-client/3.3.2/hadoop-client-3.3.2.pom > > > (11 kB at 20 kB/s) > > > ``` > > > there's no tests there, but it did audit the download process. FWIW, > that > > > project has switched to logback, so I now have all hadoop imports > > excluding > > > slf4j and log4j. it takes too much effort right now. > > > > > > build works. > > > > > > tested abfs and s3a storediags, all happy > > > > > > > > > > > > > > > *### google GCS against staged artifacts* > > > > > > gcs is now java 11 only, so I had to switch JVMs here. > > > > > > had to add a snapshots and staging profile, after which I could build > and > > > test. > > > > > > ``` > > > -Dhadoop.three.version=3.3.2 -Psnapshots-and-staging > > > ``` > > > two test failures were related to auth failures where the tests were > > trying > > > to raise exceptions but things failed differently > > > ``` > > > [ERROR] Failures: > > > [ERROR] > > > > > > > > > GoogleHadoopFileSystemTest.eagerInitialization_fails_withInvalidCredentialsConfiguration:122 > > > unexpected exception type thrown; expected:<java.io > > .FileNotFoundException> > > > but was:<java.lang.IllegalArgumentException> > > > [ERROR] > > > > > > > > > GoogleHadoopFileSystemTest.lazyInitialization_deleteCall_fails_withInvalidCredentialsConfiguration:100 > > > value of: throwable.getMessage() > > > expected: Failed to create GCS FS > > > but was : A JSON key file may not be specified at the same time as > > > credentials via configuration. > > > > > > ``` > > > > > > I'm not worried here. > > > > > > ran cloudstore's diagnostics against gcs. > > > > > > Nice to see they are now collecting IOStatistics on their input > streams. > > we > > > really need to get this collected through the parquet/orc libs and then > > > through the query engines. > > > > > > ``` > > > > bin/hadoop jar $CLOUDSTORE storediag gs://stevel-london/ > > > > > > ... > > > 2022-01-20 17:52:47,447 [main] INFO diag.StoreDiag > > > (StoreDurationInfo.java:<init>(56)) - Starting: Reading a file > > > gs://stevel-london/dir-9cbfc774-76ff-49c0-b216-d7800369c3e1/file > > > input stream summary: org.apache.hadoop.fs.FSDataInputStream@6cfd9a54: > > > com.google.cloud.hadoop.fs.gcs.GoogleHadoopFSInputStream@78c1372d > > > {counters=((stream_read_close_operations=1) > > > (stream_read_seek_backward_operations=0) (stream_read_total_bytes=7) > > > (stream_read_bytes=7) (stream_read_exceptions=0) > > > (stream_read_seek_operations=0) (stream_read_seek_bytes_skipped=0) > > > (stream_read_operations=3) (stream_read_bytes_backwards_on_seek=0) > > > (stream_read_seek_forward_operations=0) > > > (stream_read_operations_incomplete=1)); > > > gauges=(); > > > minimums=(); > > > maximums=(); > > > means=(); > > > } > > > ... > > > ``` > > > > > > *### source* > > > > > > once I'd done builds and tests which fetched from staging, I did a > local > > > build and test > > > > > > repeated download/validate of source tarball, unzip/untar > > > > > > build with java11. > > > > > > I've not done the test run there, because that directory tree doesn't > > have > > > the credentials, and this mornings run was good. > > > > > > altogether then: very happy. tests good, downstream libraries building > > and > > > linking. > > > > > > On Wed, 19 Jan 2022 at 17:50, Chao Sun <sunc...@apache.org> wrote: > > > > > > > Hi all, > > > > > > > > I've put together Hadoop 3.3.2 RC2 below: > > > > > > > > The RC is available at: > > > > http://people.apache.org/~sunchao/hadoop-3.3.2-RC2/ > > > > The RC tag is at: > > > > https://github.com/apache/hadoop/releases/tag/release-3.3.2-RC2 > > > > The Maven artifacts are staged at: > > > > > > https://repository.apache.org/content/repositories/orgapachehadoop-1332 > > > > > > > > You can find my public key at: > > > > https://downloads.apache.org/hadoop/common/KEYS > > > > > > > > I've done the following tests and they look good: > > > > - Ran all the unit tests > > > > - Started a single node HDFS cluster and tested a few simple commands > > > > - Ran all the tests in Spark using the RC2 artifacts > > > > > > > > Please evaluate the RC and vote, thanks! > > > > > > > > Best, > > > > Chao > > > > > > > > > >
