that's "dangerously interesting". I think you are right, and I also think it'll just be the version files which get generated
anyway, +1 binding * ran my new Hadoop-3 profile on spark (SPARK-23807), with the committer binding, then my downstream tests. All is well, provided you also have a spark hive JAR patched to accept hadoop 3 as a legitimate hadoop version. That's an ongoing issue in the Spark project. With that JAR on my CP my downstream tests were all happy (yesterday) * today the staging files seem to be missing, at least maven is unable to find them even when I turn the spark snapshots-and-staging profile on. That'll be the maven dist process at play, nothing else On 4 Apr 2018, at 04:13, Wangda Tan <wheele...@gmail.com<mailto:wheele...@gmail.com>> wrote: Hi Vinod / Arpit, I checked following versions: - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1: Jars in maven repo [1] are *always* different from jars in the binary tarball [2]: (I only checked hadoop-yarn-api-version.jar) (Following numbers are sizes of the jar) 2.6.5: - Jar in Maven: 1896185 - Jar in tarball: 1891485 2.7.5: - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf) - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef) (Even size is same, md5 is different) 2.8.3: - Jar in Maven: 2451433 - Jar in tarball: 2438975 2.9.0: - Jar in Maven: 2791477 - Jar in tarball: 2777789 3.0.1: - Jar in Maven: 2852604 - Jar in tarball: 2851373 I guess the differences come from our release process. Thanks, Wangda [1] Maven jars are downloaded from https://repository.apache.org/service/local/repositories/releases/content/org/apache/hadoop/hadoop-yarn-api/ <version>/hadoop-yarn-api-<version>.jar [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/ On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <vino...@apache.org<mailto:vino...@apache.org>> wrote: We vote on the source code. The binaries are convenience artifacts. This is what I would do - (a) Just replace both the maven jars as well as the binaries to be consistent and correct. And then (b) Give a couple more days for folks who tested on the binaries to reverify - I count one such clear vote as of now. Thanks +Vinod On Apr 3, 2018, at 3:30 PM, Wangda Tan <wheele...@gmail.com<mailto:wheele...@gmail.com>> wrote: HI Arpit, I think it won't match if we do rebuild. It should be fine as far as they're signed, correct? I don't see any policy doesn't allow this. Thanks, Wangda On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal <aagar...@hortonworks.com<mailto:aagar...@hortonworks.com>> wrote: Thanks Wangda, I see the shaded jars now. Are the repo jars required to be the same as the binary release? They don’t match right now, probably they got rebuilt. +1 (binding), modulo that remaining question. * Verified signatures * Verified checksums for source and binary artefacts * Sanity checked jars on r.a.o. * Built from source * Deployed to 3 node secure cluster with NameNode HA * Verified HDFS web UIs * Tried out HDFS shell commands * Ran sample MapReduce jobs Thanks! ---------------------------------------------------------------------- From: Wangda Tan <wheele...@gmail.com<mailto:wheele...@gmail.com>> Date: Monday, April 2, 2018 at 9:25 PM To: Arpit Agarwal <aagar...@hortonworks.com<mailto:aagar...@hortonworks.com>> Cc: Gera Shegalov <ger...@gmail.com<mailto:ger...@gmail.com>>, Sunil G <sun...@apache.org<mailto:sun...@apache.org>>, " yarn-...@hadoop.apache.org<mailto:yarn-...@hadoop.apache.org>" <yarn-...@hadoop.apache.org<mailto:yarn-...@hadoop.apache.org>>, Hdfs-dev < hdfs-dev@hadoop.apache.org<mailto:hdfs-dev@hadoop.apache.org>>, Hadoop Common <common-...@hadoop.apache.org<mailto:common-...@hadoop.apache.org>>, "mapreduce-...@hadoop.apache.org<mailto:mapreduce-...@hadoop.apache.org>" <mapreduce-...@hadoop.apache.org<mailto:mapreduce-...@hadoop.apache.org>>, Vinod Kumar Vavilapalli <vino...@apache.org<mailto:vino...@apache.org>> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1) As pointed by Arpit, the previously deployed shared jars are incorrect. Just redeployed jars and staged. @Arpit, could you please check the updated Maven repo? https://repository.apache.org/content/repositories/ orgapachehadoop-1092 Since the jars inside binary tarballs are correct ( http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't need roll another RC, just update Maven repo should be sufficient. Best, Wangda On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheele...@gmail.com> wrote: Hi Arpit, Thanks for pointing out this. I just removed all .md5 files from artifacts. I found md5 checksums still exist in .mds files and I didn't remove them from .mds file because it is generated by create-release script and Apache guidance is "should not" instead of "must not". Please let me know if you think they need to be removed as well. - Wangda On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto: aagar...@hortonworks.com<mailto:aagar...@hortonworks.com>> wrote: Thanks for putting together this RC, Wangda. The guidance from Apache is to omit MD5s, specifically: SHOULD NOT supply a MD5 checksum file (because MD5 is too broken). https://www.apache.org/dev/release-distribution#sigs-and-sums On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheele...@gmail.com> wrote: Hi Gera, It's my bad, I thought only src/bin tarball is enough. I just uploaded all other things under artifact/ to http://people.apache.org/~wangda/hadoop-3.1.0-RC1/ Please let me know if you have any other comments. Thanks, Wangda On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:ger...@gmail.com> wrote: Thanks, Wangda! There are many more artifacts in previous votes, e.g., see http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ . Among others the site tarball is missing. On Sun, Apr 1, 2018 at 11:54 PM Sunil G <mailto:sun...@apache.org> wrote: Thanks Wangda for initiating the release. I tested this RC built from source file. - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM UI. - Below feature sanity is done - Application priority - Application timeout - Intra Queue preemption with priority based - DS based affinity tests to verify placement constraints. - Tested basic NodeLabel scenarios. - Added couple of labels to few of nodes and behavior is coming correct. - Verified old UI and new YARN UI for labels. - Submitted apps to labelled cluster and it works fine. - Also performed few cli commands related to nodelabel. - Test basic HA cases and seems correct. - Tested new YARN UI . All pages are getting loaded correctly. - Sunil On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <mailto:wheele...@gmail.com> wrote: Hi folks, Thanks to the many who helped with this release since Dec 2017 [1]. We've created RC1 for Apache Hadoop 3.1.0. The artifacts are available here: http://people.apache.org/~wangda/hadoop-3.1.0-RC1 The RC tag in git is release-3.1.0-RC1. Last git commit SHA is 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d The maven artifacts are available via http://repository.apache.org<http://repository.apache.org/> at https://repository.apache.org/content/repositories/ orgapachehadoop-1090/ This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific. 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions include the first class GPU/FPGA support on YARN, Native services, Support rich placement constraints in YARN, S3-related enhancements, allow HDFS block replicas to be provided by an external storage system, etc. For 3.1.0 RC0 vote discussion, please see [3]. We’d like to use this as a starting release for 3.1.x [1], depending on how it goes, get it stabilized and potentially use a 3.1.1 in several weeks as the stable release. We have done testing with a pseudo cluster: - Ran distributed job. - GPU scheduling/isolation. - Placement constraints (intra-application anti-affinity) by using distributed shell. My +1 to start. Best, Wangda/Vinod [1] https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b c9e19e559aedc5ebedb4ca08c8@%http://3Cyarn-dev.hadoop.apache.org<http://3cyarn-dev.hadoop.apache.org/> <http://3cyarn-dev.hadoop.apache.org/>%3E [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0) AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY fixVersion ASC [3] https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7 2d4061f26f83547e41d0983ea6@%http://3Cyarn-dev.hadoop.apache.org<http://3cyarn-dev.hadoop.apache.org/> <http://3cyarn-dev.hadoop.apache.org/>%3E
