Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0
> On 6 Sep 2016, at 17:30, Andrew Wang wrote: > > Thanks everyone for voting and helping to validate alpha1. The VOTE closes > with 16 +1s, 6 of them binding PMC votes, and no -1s. > > I'll go ahead and wrap up the release, will send an announcement out likely > tomorrow once the mirrors have caught up. > > Best, > Andrew > Sorry I'm late with this; I was running all the tests last night and had to redo some today. Bandwidth issues. If I hadn't missed the vote, i'd have gone +0.5 I couldn't do enough due diligence to be confident all was well. But those bits I did do (s3, azure, openstack) are all good. -checked out the source, rebuilt locally, ran the Hadoop-aws s3a test suite against s3 ireland ,openstack against rackspace and azure against Azure. All well there. -built slider. Compilation failed there because Container has added things that Slider's mock containers don't implement. This is well within the compatibility scopes, albeit inconvenient. - I haven't done a full spark test run as don't have the time right now, and haven't been running locally be confident that all works. Sorry -I did do the SPARK-7481 cloud tests. These were *really* slow, but I was doing the hadoop trunk s3, azure and swift tests in different windows; I suspect I was just using up too much CPU, RAM and bandwidth. The windows build failed irrespective of whether I had -Pnative set or not. I think theres a script there that needs windows support. Were it a real release I'd veto it for that, as it meant I wouldn't be able to build the windows native libraries for the good of everyone. Filed: HADOOP-13586 Anyway, good to see the first 3.0 alpha out the door, let's see what surfaces in the field -Steve - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0
Thanks everyone for voting and helping to validate alpha1. The VOTE closes with 16 +1s, 6 of them binding PMC votes, and no -1s. I'll go ahead and wrap up the release, will send an announcement out likely tomorrow once the mirrors have caught up. Best, Andrew On Sat, Sep 3, 2016 at 6:55 PM, Akira Ajisaka wrote: > Thanks Andrew for the great work. > > +1 (binding) > > - Downloaded source and verified checksum and signature > - Compiled with -Pnative @ CentOS 7.2 and OpenJDK 8u101 > - Ran some MapReduce jobs successfully > - Checked Web UIs and logs > - hdfs/yarn/mapred "--daemon" option works as expected > > Regards, > Akira > > > On 9/4/16 09:06, Eric Payne wrote: > >> Thank you very much, Andrew. >> >> +1 (non-binding) >> >> - Downloaded source and built native >> - Installed on 3-node, non-secure cluster >> - Ran sleep jobs >> - Ensured preemption works as expected >> >> -Eric Payne >> >> >> - Original Message - >> From: Andrew Wang >> To: "common-...@hadoop.apache.org" ; " >> hdfs-...@hadoop.apache.org" ; " >> mapreduce-...@hadoop.apache.org" ; " >> yarn-dev@hadoop.apache.org" >> Sent: Tuesday, August 30, 2016 10:51 AM >> Subject: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0 >> >> Hi all, >> >> Thanks to the combined work of many, many contributors, here's an RC0 for >> 3.0.0-alpha1: >> >> http://home.apache.org/~wang/3.0.0-alpha1-RC0/ >> >> alpha1 is the first in a series of planned alpha releases leading up to >> GA. >> The objective is to get an artifact out to downstreams for testing and to >> iterate quickly based on their feedback. So, please keep that in mind when >> voting; hopefully most issues can be addressed by future alphas rather >> than >> future RCs. >> >> Sorry for getting this out on a Tuesday, but I'd still like this vote to >> run the normal 5 days, thus ending Saturday (9/3) at 9AM PDT. I'll extend >> if we lack the votes. >> >> Please try it out and let me know what you think. >> >> Best, >> Andrew >> >> - >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org >> >> > > - > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > >
Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0
Thanks Andrew for the great work. +1 (binding) - Downloaded source and verified checksum and signature - Compiled with -Pnative @ CentOS 7.2 and OpenJDK 8u101 - Ran some MapReduce jobs successfully - Checked Web UIs and logs - hdfs/yarn/mapred "--daemon" option works as expected Regards, Akira On 9/4/16 09:06, Eric Payne wrote: Thank you very much, Andrew. +1 (non-binding) - Downloaded source and built native - Installed on 3-node, non-secure cluster - Ran sleep jobs - Ensured preemption works as expected -Eric Payne - Original Message - From: Andrew Wang To: "common-...@hadoop.apache.org" ; "hdfs-...@hadoop.apache.org" ; "mapreduce-...@hadoop.apache.org" ; "yarn-dev@hadoop.apache.org" Sent: Tuesday, August 30, 2016 10:51 AM Subject: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0 Hi all, Thanks to the combined work of many, many contributors, here's an RC0 for 3.0.0-alpha1: http://home.apache.org/~wang/3.0.0-alpha1-RC0/ alpha1 is the first in a series of planned alpha releases leading up to GA. The objective is to get an artifact out to downstreams for testing and to iterate quickly based on their feedback. So, please keep that in mind when voting; hopefully most issues can be addressed by future alphas rather than future RCs. Sorry for getting this out on a Tuesday, but I'd still like this vote to run the normal 5 days, thus ending Saturday (9/3) at 9AM PDT. I'll extend if we lack the votes. Please try it out and let me know what you think. Best, Andrew - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0
+1(binding) * Compiling source code. * Verified checksums. * Running RM-HA cluster with ZooKeeper 3.4.8 and ran MR job with native libraries. It worked well. Again, I'm very happy to move 3.0.0 release forward :-) Thanks, Andrew! FYI: > In fact, I hit an error similar to Eric's one regardless of configuring yarn.app.mapreduce.am.env and mapreduce.admin.user.env... does anyone hit same error? This was because of completely my wrong configuration. Please ignore this. Best, - Tsuyoshi On Sat, Sep 3, 2016 at 1:53 AM, Tsuyoshi Ozawa wrote: > Thanks for making this forward hardly, Andrew! > > In fact, I hit an error similar to Eric's one regardless of > configuring yarn.app.mapreduce.am.env and mapreduce.admin.user.env... > does anyone hit same error? > > > 2016-09-02 16:49:35,223 INFO mapreduce.Job: The url to track the job: > http://ip-172-31-7-124.ap-northeast-1.compute.internal:23188/proxy/application_1472832909785_0013/ > > 2016-09-02 16:49:35,224 INFO mapreduce.Job: Running job: > job_1472832909785_0013 > > 2016-09-02 16:49:48,282 INFO mapreduce.Job: Job job_1472832909785_0013 > running in uber mode : false > > 2016-09-02 16:49:48,283 INFO mapreduce.Job: map 0% reduce 0% > > 2016-09-02 16:49:48,293 INFO mapreduce.Job: Job job_1472832909785_0013 > failed with state FAILED due to: Application > application_1472832909785_0013 failed 2 times due to AM Container for > appattempt_1472832909785_0013_02 exited with exitCode: 1 > > Failing this attempt.Diagnostics: Exception from container-launch. > > Container id: container_e02_1472832909785_0013_02_01 > > Exit code: 1 > > Stack trace: ExitCodeException exitCode=1: > > at org.apache.hadoop.util.Shell.runCommand(Shell.java:974) > > at org.apache.hadoop.util.Shell.run(Shell.java:878) > > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1172) > > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > at java.lang.Thread.run(Thread.java:745) > > Best, > - Tsuyoshi > > On Wed, Aug 31, 2016 at 12:51 AM, Andrew Wang > wrote: >> Hi all, >> >> Thanks to the combined work of many, many contributors, here's an RC0 for >> 3.0.0-alpha1: >> >> http://home.apache.org/~wang/3.0.0-alpha1-RC0/ >> >> alpha1 is the first in a series of planned alpha releases leading up to GA. >> The objective is to get an artifact out to downstreams for testing and to >> iterate quickly based on their feedback. So, please keep that in mind when >> voting; hopefully most issues can be addressed by future alphas rather than >> future RCs. >> >> Sorry for getting this out on a Tuesday, but I'd still like this vote to >> run the normal 5 days, thus ending Saturday (9/3) at 9AM PDT. I'll extend >> if we lack the votes. >> >> Please try it out and let me know what you think. >> >> Best, >> Andrew - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
RE: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0
Thanks Sammi. My non-binding +1 to make the release candidate. Regards, Kai -Original Message- From: Chen, Sammi Sent: Friday, September 02, 2016 4:59 PM To: Zheng, Kai ; Andrew Wang ; Arun Suresh Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; Chen, Sammi Subject: RE: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0 +1 (non-binding). Thanks for driving this Andrew! * Download and built from source. * Setup a 10 node cluster (1 name node + 9 data nodes) * Verified normal HDFS file put/get operation with 3x replication * With 2 data nodes failure, verified HDFS file put/get operation with 3x replication, file integrity is OK * Enable Erasure Code policy "RS-DEFAULT-6-3-64k", verified HDFS file put/get operation * Enable Erasure Code policy "RS-DEFAULT-6-3-64k", with 3 data nodes failure, verified HDFS file put/get operation, file integrity is OK Cheers -Sammi -Original Message- From: Zheng, Kai Sent: Friday, September 02, 2016 3:25 PM To: Chen, Sammi Subject: FW: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0 Hi Sammi, Could you help provide our feedback? I know you did lots of tests. Thanks! Regards, Kai -Original Message- From: Arun Suresh [mailto:asur...@apache.org] Sent: Friday, September 02, 2016 11:33 AM To: Andrew Wang Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org Subject: Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0 +1 (binding). Thanks for driving this Andrew.. * Download and built from source. * Setup a 5 mode cluster. * Verified that MR works with opportunistic containers * Verified that the AMRMClient supports 'allocationRequestId' Cheers -Arun On Thu, Sep 1, 2016 at 4:31 PM, Aaron Fabbri wrote: > +1, non-binding. > > I built everything on OS X and ran the s3a contract tests successfully: > > mvn test -Dtest=org.apache.hadoop.fs.contract.s3a.\* > > ... > > Results : > > > Tests run: 78, Failures: 0, Errors: 0, Skipped: 1 > > > [INFO] > -- > -- > > [INFO] BUILD SUCCESS > > [INFO] > -- > -- > > On Thu, Sep 1, 2016 at 3:39 PM, Andrew Wang > wrote: > > > Good point Allen, I forgot about `hadoop version`. Since it's > > populated > by > > a version-info.properties file, people can always cat that file. > > > > On Thu, Sep 1, 2016 at 3:21 PM, Allen Wittenauer < > a...@effectivemachines.com > > > > > wrote: > > > > > > > > > On Sep 1, 2016, at 3:18 PM, Allen Wittenauer < > a...@effectivemachines.com > > > > > > wrote: > > > > > > > > > > > >> On Sep 1, 2016, at 2:57 PM, Andrew Wang > > > >> > > > wrote: > > > >> > > > >> Steve requested a git hash for this release. This led us into a > brief > > > >> discussion of our use of git tags, wherein we realized that > > > >> although release tags are immutable (start with "rel/"), RC tags are > > > >> not. > This > > is > > > >> based on the HowToRelease instructions. > > > > > > > > We should probably embed the git hash in one of the files > > > > that > > > gets gpg signed. That's an easy change to create-release. > > > > > > > > > (Well, one more easily accessible than 'hadoop version') > > > - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0
I'm sorry, very big typo I did... s/hardly/very hard/ Best - Tsuyoshi On Sat, Sep 3, 2016 at 1:53 AM, Tsuyoshi Ozawa wrote: > Thanks for making this forward hardly, Andrew! > > In fact, I hit an error similar to Eric's one regardless of > configuring yarn.app.mapreduce.am.env and mapreduce.admin.user.env... > does anyone hit same error? > > > 2016-09-02 16:49:35,223 INFO mapreduce.Job: The url to track the job: > http://ip-172-31-7-124.ap-northeast-1.compute.internal:23188/proxy/application_1472832909785_0013/ > > 2016-09-02 16:49:35,224 INFO mapreduce.Job: Running job: > job_1472832909785_0013 > > 2016-09-02 16:49:48,282 INFO mapreduce.Job: Job job_1472832909785_0013 > running in uber mode : false > > 2016-09-02 16:49:48,283 INFO mapreduce.Job: map 0% reduce 0% > > 2016-09-02 16:49:48,293 INFO mapreduce.Job: Job job_1472832909785_0013 > failed with state FAILED due to: Application > application_1472832909785_0013 failed 2 times due to AM Container for > appattempt_1472832909785_0013_02 exited with exitCode: 1 > > Failing this attempt.Diagnostics: Exception from container-launch. > > Container id: container_e02_1472832909785_0013_02_01 > > Exit code: 1 > > Stack trace: ExitCodeException exitCode=1: > > at org.apache.hadoop.util.Shell.runCommand(Shell.java:974) > > at org.apache.hadoop.util.Shell.run(Shell.java:878) > > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1172) > > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > at java.lang.Thread.run(Thread.java:745) > > Best, > - Tsuyoshi > > On Wed, Aug 31, 2016 at 12:51 AM, Andrew Wang > wrote: >> Hi all, >> >> Thanks to the combined work of many, many contributors, here's an RC0 for >> 3.0.0-alpha1: >> >> http://home.apache.org/~wang/3.0.0-alpha1-RC0/ >> >> alpha1 is the first in a series of planned alpha releases leading up to GA. >> The objective is to get an artifact out to downstreams for testing and to >> iterate quickly based on their feedback. So, please keep that in mind when >> voting; hopefully most issues can be addressed by future alphas rather than >> future RCs. >> >> Sorry for getting this out on a Tuesday, but I'd still like this vote to >> run the normal 5 days, thus ending Saturday (9/3) at 9AM PDT. I'll extend >> if we lack the votes. >> >> Please try it out and let me know what you think. >> >> Best, >> Andrew - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0
Thanks for making this forward hardly, Andrew! In fact, I hit an error similar to Eric's one regardless of configuring yarn.app.mapreduce.am.env and mapreduce.admin.user.env... does anyone hit same error? 2016-09-02 16:49:35,223 INFO mapreduce.Job: The url to track the job: http://ip-172-31-7-124.ap-northeast-1.compute.internal:23188/proxy/application_1472832909785_0013/ 2016-09-02 16:49:35,224 INFO mapreduce.Job: Running job: job_1472832909785_0013 2016-09-02 16:49:48,282 INFO mapreduce.Job: Job job_1472832909785_0013 running in uber mode : false 2016-09-02 16:49:48,283 INFO mapreduce.Job: map 0% reduce 0% 2016-09-02 16:49:48,293 INFO mapreduce.Job: Job job_1472832909785_0013 failed with state FAILED due to: Application application_1472832909785_0013 failed 2 times due to AM Container for appattempt_1472832909785_0013_02 exited with exitCode: 1 Failing this attempt.Diagnostics: Exception from container-launch. Container id: container_e02_1472832909785_0013_02_01 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:974) at org.apache.hadoop.util.Shell.run(Shell.java:878) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1172) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:237) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:419) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:284) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:88) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Best, - Tsuyoshi On Wed, Aug 31, 2016 at 12:51 AM, Andrew Wang wrote: > Hi all, > > Thanks to the combined work of many, many contributors, here's an RC0 for > 3.0.0-alpha1: > > http://home.apache.org/~wang/3.0.0-alpha1-RC0/ > > alpha1 is the first in a series of planned alpha releases leading up to GA. > The objective is to get an artifact out to downstreams for testing and to > iterate quickly based on their feedback. So, please keep that in mind when > voting; hopefully most issues can be addressed by future alphas rather than > future RCs. > > Sorry for getting this out on a Tuesday, but I'd still like this vote to > run the normal 5 days, thus ending Saturday (9/3) at 9AM PDT. I'll extend > if we lack the votes. > > Please try it out and let me know what you think. > > Best, > Andrew - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0
+1( non-binding) * Successfully downloaded and built from source. * Deployed to single node cluster. * Ran Sleep and Wordcount jobs. Thanks Andrew for the effort! Regards,Kuhu On Thursday, September 1, 2016 10:32 PM, Arun Suresh wrote: +1 (binding). Thanks for driving this Andrew.. * Download and built from source. * Setup a 5 mode cluster. * Verified that MR works with opportunistic containers * Verified that the AMRMClient supports 'allocationRequestId' Cheers -Arun On Thu, Sep 1, 2016 at 4:31 PM, Aaron Fabbri wrote: > +1, non-binding. > > I built everything on OS X and ran the s3a contract tests successfully: > > mvn test -Dtest=org.apache.hadoop.fs.contract.s3a.\* > > ... > > Results : > > > Tests run: 78, Failures: 0, Errors: 0, Skipped: 1 > > > [INFO] > > > [INFO] BUILD SUCCESS > > [INFO] > > > On Thu, Sep 1, 2016 at 3:39 PM, Andrew Wang > wrote: > > > Good point Allen, I forgot about `hadoop version`. Since it's populated > by > > a version-info.properties file, people can always cat that file. > > > > On Thu, Sep 1, 2016 at 3:21 PM, Allen Wittenauer < > a...@effectivemachines.com > > > > > wrote: > > > > > > > > > On Sep 1, 2016, at 3:18 PM, Allen Wittenauer < > a...@effectivemachines.com > > > > > > wrote: > > > > > > > > > > > >> On Sep 1, 2016, at 2:57 PM, Andrew Wang > > > wrote: > > > >> > > > >> Steve requested a git hash for this release. This led us into a > brief > > > >> discussion of our use of git tags, wherein we realized that although > > > >> release tags are immutable (start with "rel/"), RC tags are not. > This > > is > > > >> based on the HowToRelease instructions. > > > > > > > > We should probably embed the git hash in one of the files that > > > gets gpg signed. That's an easy change to create-release. > > > > > > > > > (Well, one more easily accessible than 'hadoop version') > > >
Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0
+1 (binding). Thanks for driving this Andrew.. * Download and built from source. * Setup a 5 mode cluster. * Verified that MR works with opportunistic containers * Verified that the AMRMClient supports 'allocationRequestId' Cheers -Arun On Thu, Sep 1, 2016 at 4:31 PM, Aaron Fabbri wrote: > +1, non-binding. > > I built everything on OS X and ran the s3a contract tests successfully: > > mvn test -Dtest=org.apache.hadoop.fs.contract.s3a.\* > > ... > > Results : > > > Tests run: 78, Failures: 0, Errors: 0, Skipped: 1 > > > [INFO] > > > [INFO] BUILD SUCCESS > > [INFO] > > > On Thu, Sep 1, 2016 at 3:39 PM, Andrew Wang > wrote: > > > Good point Allen, I forgot about `hadoop version`. Since it's populated > by > > a version-info.properties file, people can always cat that file. > > > > On Thu, Sep 1, 2016 at 3:21 PM, Allen Wittenauer < > a...@effectivemachines.com > > > > > wrote: > > > > > > > > > On Sep 1, 2016, at 3:18 PM, Allen Wittenauer < > a...@effectivemachines.com > > > > > > wrote: > > > > > > > > > > > >> On Sep 1, 2016, at 2:57 PM, Andrew Wang > > > wrote: > > > >> > > > >> Steve requested a git hash for this release. This led us into a > brief > > > >> discussion of our use of git tags, wherein we realized that although > > > >> release tags are immutable (start with "rel/"), RC tags are not. > This > > is > > > >> based on the HowToRelease instructions. > > > > > > > > We should probably embed the git hash in one of the files that > > > gets gpg signed. That's an easy change to create-release. > > > > > > > > > (Well, one more easily accessible than 'hadoop version') > > >
Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0
Good point Allen, I forgot about `hadoop version`. Since it's populated by a version-info.properties file, people can always cat that file. On Thu, Sep 1, 2016 at 3:21 PM, Allen Wittenauer wrote: > > > On Sep 1, 2016, at 3:18 PM, Allen Wittenauer > wrote: > > > > > >> On Sep 1, 2016, at 2:57 PM, Andrew Wang > wrote: > >> > >> Steve requested a git hash for this release. This led us into a brief > >> discussion of our use of git tags, wherein we realized that although > >> release tags are immutable (start with "rel/"), RC tags are not. This is > >> based on the HowToRelease instructions. > > > > We should probably embed the git hash in one of the files that > gets gpg signed. That's an easy change to create-release. > > > (Well, one more easily accessible than 'hadoop version')
Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0
> On Sep 1, 2016, at 3:18 PM, Allen Wittenauer > wrote: > > >> On Sep 1, 2016, at 2:57 PM, Andrew Wang wrote: >> >> Steve requested a git hash for this release. This led us into a brief >> discussion of our use of git tags, wherein we realized that although >> release tags are immutable (start with "rel/"), RC tags are not. This is >> based on the HowToRelease instructions. > > We should probably embed the git hash in one of the files that gets gpg > signed. That's an easy change to create-release. (Well, one more easily accessible than 'hadoop version') - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0
> On Sep 1, 2016, at 2:57 PM, Andrew Wang wrote: > > Steve requested a git hash for this release. This led us into a brief > discussion of our use of git tags, wherein we realized that although > release tags are immutable (start with "rel/"), RC tags are not. This is > based on the HowToRelease instructions. We should probably embed the git hash in one of the files that gets gpg signed. That's an easy change to create-release. - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0
Steve requested a git hash for this release. This led us into a brief discussion of our use of git tags, wherein we realized that although release tags are immutable (start with "rel/"), RC tags are not. This is based on the HowToRelease instructions. I asked in infra.chat about this, and filed a JIRA per their request. I'll update the tags and release instructions once we have guidance. https://issues.apache.org/jira/browse/INFRA-12552 For now though, here's the tag I pushed (email is immutable): object a990d2ebcd6de5d7dc2d3684930759b0f0ea4dc3 type commit tag release-3.0.0-alpha1-RC0 tagger Andrew Wang 1472541776 -0700 Release candidate - 3.0.0-alpha1-RC0 gpg: Signature made Tue 30 Aug 2016 12:22:56 AM PDT using RSA key ID 7501105C gpg: Good signature from "Andrew Wang (CODE SIGNING KEY) < andrew.w...@cloudera.com>" gpg: aka "Andrew Wang (CODE SIGNING KEY) " On Tue, Aug 30, 2016 at 8:51 AM, Andrew Wang wrote: > Hi all, > > Thanks to the combined work of many, many contributors, here's an RC0 for > 3.0.0-alpha1: > > http://home.apache.org/~wang/3.0.0-alpha1-RC0/ > > alpha1 is the first in a series of planned alpha releases leading up to > GA. The objective is to get an artifact out to downstreams for testing and > to iterate quickly based on their feedback. So, please keep that in mind > when voting; hopefully most issues can be addressed by future alphas rather > than future RCs. > > Sorry for getting this out on a Tuesday, but I'd still like this vote to > run the normal 5 days, thus ending Saturday (9/3) at 9AM PDT. I'll extend > if we lack the votes. > > Please try it out and let me know what you think. > > Best, > Andrew >
Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0
+1 (non-binding) Thanks Andrew for putting this up! Super excited to see a Hadoop 3 RC. Verifications done on OS X: - Verified md5 on all files - Spot checked release notes and changes - Built from source - Verified LICENSE and NOTICE are correctly contained in the jars built - Started pseudo-distributed HDFS, verified basic operations work. - Started KMS service, verified basic KMS operations work, a simple encryption zone works. - Sanity checked NN webui. Best, -Xiao On Wed, Aug 31, 2016 at 9:25 PM, Rakesh Radhakrishnan wrote: > Thanks for getting this out. > > +1 (non-binding) > > - downloaded and built tarball from source > - deployed HDFS-HA cluster and tested few EC file operations > - executed few hdfs commands including EC commands > - viewed basic UI > - ran some of the sample jobs > > > Best Regards, > Rakesh > Intel > > On Thu, Sep 1, 2016 at 6:19 AM, John Zhuge wrote: > > > +1 (non-binding) > > > > - Build source with Java 1.8.0_101 on Centos 6.6 without native > > - Verify license and notice using the shell script in HADOOP-13374 > > - Deploy a pseudo cluster > > - Run basic dfs, distcp, ACL, webhdfs commands > > - Run MapReduce workcount and pi examples > > - Run balancer > > > > Thanks, > > John > > > > John Zhuge > > Software Engineer, Cloudera > > > > On Wed, Aug 31, 2016 at 11:46 AM, Gangumalla, Uma < > > uma.ganguma...@intel.com> > > wrote: > > > > > +1 (binding). > > > > > > Overall it¹s a great effort, Andrew. Thank you for putting all the > > energy. > > > > > > Downloaded and built. > > > Ran some sample jobs. > > > > > > I would love to see all this efforts will lead to get the GA from > Hadoop > > > 3.X soon. > > > > > > Regards, > > > Uma > > > > > > > > > On 8/30/16, 8:51 AM, "Andrew Wang" wrote: > > > > > > >Hi all, > > > > > > > >Thanks to the combined work of many, many contributors, here's an RC0 > > for > > > >3.0.0-alpha1: > > > > > > > >http://home.apache.org/~wang/3.0.0-alpha1-RC0/ > > > > > > > >alpha1 is the first in a series of planned alpha releases leading up > to > > > >GA. > > > >The objective is to get an artifact out to downstreams for testing and > > to > > > >iterate quickly based on their feedback. So, please keep that in mind > > when > > > >voting; hopefully most issues can be addressed by future alphas rather > > > >than > > > >future RCs. > > > > > > > >Sorry for getting this out on a Tuesday, but I'd still like this vote > to > > > >run the normal 5 days, thus ending Saturday (9/3) at 9AM PDT. I'll > > extend > > > >if we lack the votes. > > > > > > > >Please try it out and let me know what you think. > > > > > > > >Best, > > > >Andrew > > > > > > > > > - > > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > > > > > > > >
Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0
+1 (binding) from me. Downloaded the source, built from source, set up a pseudo cluster, and ran a few of the sample jobs. Thanks a lot for doing all this release work, Andrew. -- Aaron T. Myers Software Engineer, Cloudera On Tue, Aug 30, 2016 at 8:51 AM, Andrew Wang wrote: > Hi all, > > Thanks to the combined work of many, many contributors, here's an RC0 for > 3.0.0-alpha1: > > http://home.apache.org/~wang/3.0.0-alpha1-RC0/ > > alpha1 is the first in a series of planned alpha releases leading up to GA. > The objective is to get an artifact out to downstreams for testing and to > iterate quickly based on their feedback. So, please keep that in mind when > voting; hopefully most issues can be addressed by future alphas rather than > future RCs. > > Sorry for getting this out on a Tuesday, but I'd still like this vote to > run the normal 5 days, thus ending Saturday (9/3) at 9AM PDT. I'll extend > if we lack the votes. > > Please try it out and let me know what you think. > > Best, > Andrew >
Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0
It's also the key Andrew has in the project's KEYS file: http://www.apache.org/dist/hadoop/common/KEYS On Tue, Aug 30, 2016 at 4:12 PM, Andrew Wang wrote: > Hi Eric, thanks for trying this out, > > I tried this gpg command to get my key, seemed to work: > > # gpg --keyserver pgp.mit.edu --recv-keys 7501105C > gpg: requesting key 7501105C from hkp server pgp.mit.edu > gpg: /root/.gnupg/trustdb.gpg: trustdb created > gpg: key 7501105C: public key "Andrew Wang (CODE SIGNING KEY) < > andrew.w...@cloudera.com>" imported > gpg: no ultimately trusted keys found > gpg: Total number processed: 1 > gpg: imported: 1 (RSA: 1) > > Also found via search: > http://pgp.mit.edu/pks/lookup?search=wang%40apache.org&op=index > > > On Tue, Aug 30, 2016 at 2:06 PM, Eric Badger wrote: > >> I don't know why my email client keeps getting rid of all of my spacing. >> Resending the same email so that it is actually legible... >> >> All on OSX 10.11.6: >> - Verified the hashes. However, Andrew, I don't know where to find your >> public key, so I wasn't able to verify that they were signed by you. >> - Built from source >> - Deployed a pseudo-distributed clusterRan a few sample jobs >> - Poked around the RM UI >> - Poked around the attached website locally via the tarball >> >> >> I did find one odd thing, though. It could be a misconfiguration on my >> system, but I've never had this problem before with other releases (though >> I deal almost exclusively in 2.x and so I imagine things might be >> different). When I run a sleep job, I do not see any >> diagnostics/logs/counters printed out by the client. Initially I ran the >> job like I would on 2.7 and it failed (because I had not set >> yarn.app.mapreduce.am.env and mapreduce.admin.user.env), but I didn't see >> anything until I looked at the RM UI. There I was able to see all of the >> logs for the failed job and diagnose the issue. Then, once I fixed my >> parameters and ran the job again, I still didn't see any >> diagnostics/logs/counters. >> >> >> ebadger@foo: env | grep HADOOP >> HADOOP_HOME=/Users/ebadger/Downloads/hadoop-3.0.0-alpha1- >> src/hadoop-dist/target/hadoop-3.0.0-alpha1/ >> HADOOP_CONF_DIR=/Users/ebadger/conf >> ebadger@foo: $HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/share/hadoop/ >> mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-alpha1-tests.jar sleep >> -Dyarn.app.mapreduce.am.env="HADOOP_MAPRED_HOME=$HADOOP_HOME" >> -Dmapreduce.admin.user.env="HADOOP_MAPRED_HOME=$HADOOP_HOME" -mt 1 -rt 1 >> -m 1 -r 1 >> WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete. >> ebadger@foo: >> >> >> After running the above command, the RM UI showed a successful job, but as >> you can see, I did not have anything printed onto the command line. >> Hopefully this is just a misconfiguration on my part, but I figured that I >> would point it out just in case. >> >> >> Thanks, >> >> >> Eric >> >> >> >> On Tuesday, August 30, 2016 4:00 PM, Eric Badger >> wrote: >> >> >> >> All on OSX 10.11.6: >> Verified the hashes. However, Andrew, I don't know where to find your >> public key, so I wasn't able to verify that they were signed by you.Built >> from sourceDeployed a pseudo-distributed clusterRan a few sample jobsPoked >> around the RM UIPoked around the attached website locally via the tarball >> I did find one odd thing, though. It could be a misconfiguration on my >> system, but I've never had this problem before with other releases (though >> I deal almost exclusively in 2.x and so I imagine things might be >> different). When I run a sleep job, I do not see any >> diagnostics/logs/counters printed out by the client. Initially I ran the >> job like I would on 2.7 and it failed (because I had not set >> yarn.app.mapreduce.am.env and mapreduce.admin.user.env), but I didn't see >> anything until I looked at the RM UI. There I was able to see all of the >> logs for the failed job and diagnose the issue. Then, once I fixed my >> parameters and ran the job again, I still didn't see any >> diagnostics/logs/counters. >> ebadger@foo: env | grep HADOOPHADOOP_HOME=/Users/ >> ebadger/Downloads/hadoop-3.0.0-alpha1-src/hadoop-dist/ >> target/hadoop-3.0.0-alpha1/HADOOP_CONF_DIR=/Users/ebadger/confebadger@foo: >> $HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/share/hadoop/ >> mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-alpha1-tests.jar sleep >> -Dyarn.app.mapreduce.am.env="HADOOP_MAPRED_HOME=$HADOOP_HOME" >> -Dmapreduce.admin.user.env="HADOOP_MAPRED_HOME=$HADOOP_HOME" -mt 1 -rt 1 >> -m 1 -r 1WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be >> incomplete.ebadger@foo: >> After running the above command, the RM UI showed a successful job, but as >> you can see, I did not have anything printed onto the command line. >> Hopefully this is just a misconfiguration on my part, but I figured that I >> would point it out just in case. >> Thanks, >> Eric >> >> >> >> On Tuesday, August 30, 2016 12:58 PM, Andrew Wang < >> andrew.w...@cloudera.com> wrote:
Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0
+1 (non-binding) Did the following on 7 RHEL 6.6 servers - Downloaded and built from source - Downloaded and verified checksum of the binary tar.gz file - Setup a cluster with 1 NN and 6 DNs - Tried regular HDFS commands - Tried EC commands (listPolicies, getPolicy, setPolicy), they work fine - Verified that with a 3-2 policy, 1.67x capacity is used. Below is the output after copying the binary tar.gz file into an EC folder. The file is 318MB. Configured Capacity: 3221225472 (3 GB) Present Capacity: 3215348743 (2.99 GB) DFS Remaining: 2655666176 (2.47 GB) DFS Used: 559682567 (533.75 MB) Thanks Allen for clarifying on the markdown files. I also verified the site html files (content of the index.html, randomly selected some links). On Tue, Aug 30, 2016 at 2:20 PM Eric Badger wrote: > Well that's embarrassing. I had accidentally slightly renamed my > log4j.properties file in my conf directory, so it was there, just not being > read. Apologies for the unnecessary spam. With this and the public key from > Andrew, I give my non-binding +1. > > Eric > > > > On Tuesday, August 30, 2016 4:11 PM, Allen Wittenauer < > a...@effectivemachines.com> wrote: > > > > On Aug 30, 2016, at 2:06 PM, Eric Badger > wrote: > > > > > > WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be > incomplete. > > ^^ > > > > > > After running the above command, the RM UI showed a successful job, but > as you can see, I did not have anything printed onto the command line. > Hopefully this is just a misconfiguration on my part, but I figured that I > would point it out just in case. > > > It gave you a very important message in the output ... > > - > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: common-dev-h...@hadoop.apache.org > >
Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0
> On Aug 30, 2016, at 2:20 PM, Eric Badger wrote: > > Well that's embarrassing. I had accidentally slightly renamed my > log4j.properties file in my conf directory, so it was there, just not being > read. Nah. You were just testing out the shell rewrite's ability to detect a common error. ;) BTW, something else.. instead of doing env|grep HADOOP, you can do 'hadoop envvars' to get most of the good stuff. - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0
Well that's embarrassing. I had accidentally slightly renamed my log4j.properties file in my conf directory, so it was there, just not being read. Apologies for the unnecessary spam. With this and the public key from Andrew, I give my non-binding +1. Eric On Tuesday, August 30, 2016 4:11 PM, Allen Wittenauer wrote: > On Aug 30, 2016, at 2:06 PM, Eric Badger > wrote: > > > WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete. ^^ > > After running the above command, the RM UI showed a successful job, but as > you can see, I did not have anything printed onto the command line. Hopefully > this is just a misconfiguration on my part, but I figured that I would point > it out just in case. It gave you a very important message in the output ... - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0
Hi Eric, thanks for trying this out, I tried this gpg command to get my key, seemed to work: # gpg --keyserver pgp.mit.edu --recv-keys 7501105C gpg: requesting key 7501105C from hkp server pgp.mit.edu gpg: /root/.gnupg/trustdb.gpg: trustdb created gpg: key 7501105C: public key "Andrew Wang (CODE SIGNING KEY) < andrew.w...@cloudera.com>" imported gpg: no ultimately trusted keys found gpg: Total number processed: 1 gpg: imported: 1 (RSA: 1) Also found via search: http://pgp.mit.edu/pks/lookup?search=wang%40apache.org&op=index On Tue, Aug 30, 2016 at 2:06 PM, Eric Badger wrote: > I don't know why my email client keeps getting rid of all of my spacing. > Resending the same email so that it is actually legible... > > All on OSX 10.11.6: > - Verified the hashes. However, Andrew, I don't know where to find your > public key, so I wasn't able to verify that they were signed by you. > - Built from source > - Deployed a pseudo-distributed clusterRan a few sample jobs > - Poked around the RM UI > - Poked around the attached website locally via the tarball > > > I did find one odd thing, though. It could be a misconfiguration on my > system, but I've never had this problem before with other releases (though > I deal almost exclusively in 2.x and so I imagine things might be > different). When I run a sleep job, I do not see any > diagnostics/logs/counters printed out by the client. Initially I ran the > job like I would on 2.7 and it failed (because I had not set > yarn.app.mapreduce.am.env and mapreduce.admin.user.env), but I didn't see > anything until I looked at the RM UI. There I was able to see all of the > logs for the failed job and diagnose the issue. Then, once I fixed my > parameters and ran the job again, I still didn't see any > diagnostics/logs/counters. > > > ebadger@foo: env | grep HADOOP > HADOOP_HOME=/Users/ebadger/Downloads/hadoop-3.0.0-alpha1- > src/hadoop-dist/target/hadoop-3.0.0-alpha1/ > HADOOP_CONF_DIR=/Users/ebadger/conf > ebadger@foo: $HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/share/hadoop/ > mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-alpha1-tests.jar sleep > -Dyarn.app.mapreduce.am.env="HADOOP_MAPRED_HOME=$HADOOP_HOME" > -Dmapreduce.admin.user.env="HADOOP_MAPRED_HOME=$HADOOP_HOME" -mt 1 -rt 1 > -m 1 -r 1 > WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete. > ebadger@foo: > > > After running the above command, the RM UI showed a successful job, but as > you can see, I did not have anything printed onto the command line. > Hopefully this is just a misconfiguration on my part, but I figured that I > would point it out just in case. > > > Thanks, > > > Eric > > > > On Tuesday, August 30, 2016 4:00 PM, Eric Badger > wrote: > > > > All on OSX 10.11.6: > Verified the hashes. However, Andrew, I don't know where to find your > public key, so I wasn't able to verify that they were signed by you.Built > from sourceDeployed a pseudo-distributed clusterRan a few sample jobsPoked > around the RM UIPoked around the attached website locally via the tarball > I did find one odd thing, though. It could be a misconfiguration on my > system, but I've never had this problem before with other releases (though > I deal almost exclusively in 2.x and so I imagine things might be > different). When I run a sleep job, I do not see any > diagnostics/logs/counters printed out by the client. Initially I ran the > job like I would on 2.7 and it failed (because I had not set > yarn.app.mapreduce.am.env and mapreduce.admin.user.env), but I didn't see > anything until I looked at the RM UI. There I was able to see all of the > logs for the failed job and diagnose the issue. Then, once I fixed my > parameters and ran the job again, I still didn't see any > diagnostics/logs/counters. > ebadger@foo: env | grep HADOOPHADOOP_HOME=/Users/ > ebadger/Downloads/hadoop-3.0.0-alpha1-src/hadoop-dist/ > target/hadoop-3.0.0-alpha1/HADOOP_CONF_DIR=/Users/ebadger/confebadger@foo: > $HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/share/hadoop/ > mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-alpha1-tests.jar sleep > -Dyarn.app.mapreduce.am.env="HADOOP_MAPRED_HOME=$HADOOP_HOME" > -Dmapreduce.admin.user.env="HADOOP_MAPRED_HOME=$HADOOP_HOME" -mt 1 -rt 1 > -m 1 -r 1WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be > incomplete.ebadger@foo: > After running the above command, the RM UI showed a successful job, but as > you can see, I did not have anything printed onto the command line. > Hopefully this is just a misconfiguration on my part, but I figured that I > would point it out just in case. > Thanks, > Eric > > > > On Tuesday, August 30, 2016 12:58 PM, Andrew Wang < > andrew.w...@cloudera.com> wrote: > > > I'll put my own +1 on it: > > * Built from source > * Started pseudo cluster and ran Pi job successfully > > On Tue, Aug 30, 2016 at 10:17 AM, Zhe Zhang wrote: > > > > > Thanks Andrew for the great work! It's really exciting to finally see a > > Hadoop 3 RC. > > > > I noticed
Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0
> On Aug 30, 2016, at 2:06 PM, Eric Badger > wrote: > > > WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete. ^^ > > After running the above command, the RM UI showed a successful job, but as > you can see, I did not have anything printed onto the command line. Hopefully > this is just a misconfiguration on my part, but I figured that I would point > it out just in case. It gave you a very important message in the output ... - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0
I don't know why my email client keeps getting rid of all of my spacing. Resending the same email so that it is actually legible... All on OSX 10.11.6: - Verified the hashes. However, Andrew, I don't know where to find your public key, so I wasn't able to verify that they were signed by you. - Built from source - Deployed a pseudo-distributed clusterRan a few sample jobs - Poked around the RM UI - Poked around the attached website locally via the tarball I did find one odd thing, though. It could be a misconfiguration on my system, but I've never had this problem before with other releases (though I deal almost exclusively in 2.x and so I imagine things might be different). When I run a sleep job, I do not see any diagnostics/logs/counters printed out by the client. Initially I ran the job like I would on 2.7 and it failed (because I had not set yarn.app.mapreduce.am.env and mapreduce.admin.user.env), but I didn't see anything until I looked at the RM UI. There I was able to see all of the logs for the failed job and diagnose the issue. Then, once I fixed my parameters and ran the job again, I still didn't see any diagnostics/logs/counters. ebadger@foo: env | grep HADOOP HADOOP_HOME=/Users/ebadger/Downloads/hadoop-3.0.0-alpha1-src/hadoop-dist/target/hadoop-3.0.0-alpha1/ HADOOP_CONF_DIR=/Users/ebadger/conf ebadger@foo: $HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-alpha1-tests.jar sleep -Dyarn.app.mapreduce.am.env="HADOOP_MAPRED_HOME=$HADOOP_HOME" -Dmapreduce.admin.user.env="HADOOP_MAPRED_HOME=$HADOOP_HOME" -mt 1 -rt 1 -m 1 -r 1 WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete. ebadger@foo: After running the above command, the RM UI showed a successful job, but as you can see, I did not have anything printed onto the command line. Hopefully this is just a misconfiguration on my part, but I figured that I would point it out just in case. Thanks, Eric On Tuesday, August 30, 2016 4:00 PM, Eric Badger wrote: All on OSX 10.11.6: Verified the hashes. However, Andrew, I don't know where to find your public key, so I wasn't able to verify that they were signed by you.Built from sourceDeployed a pseudo-distributed clusterRan a few sample jobsPoked around the RM UIPoked around the attached website locally via the tarball I did find one odd thing, though. It could be a misconfiguration on my system, but I've never had this problem before with other releases (though I deal almost exclusively in 2.x and so I imagine things might be different). When I run a sleep job, I do not see any diagnostics/logs/counters printed out by the client. Initially I ran the job like I would on 2.7 and it failed (because I had not set yarn.app.mapreduce.am.env and mapreduce.admin.user.env), but I didn't see anything until I looked at the RM UI. There I was able to see all of the logs for the failed job and diagnose the issue. Then, once I fixed my parameters and ran the job again, I still didn't see any diagnostics/logs/counters. ebadger@foo: env | grep HADOOPHADOOP_HOME=/Users/ebadger/Downloads/hadoop-3.0.0-alpha1-src/hadoop-dist/target/hadoop-3.0.0-alpha1/HADOOP_CONF_DIR=/Users/ebadger/confebadger@foo: $HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-alpha1-tests.jar sleep -Dyarn.app.mapreduce.am.env="HADOOP_MAPRED_HOME=$HADOOP_HOME" -Dmapreduce.admin.user.env="HADOOP_MAPRED_HOME=$HADOOP_HOME" -mt 1 -rt 1 -m 1 -r 1WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.ebadger@foo: After running the above command, the RM UI showed a successful job, but as you can see, I did not have anything printed onto the command line. Hopefully this is just a misconfiguration on my part, but I figured that I would point it out just in case. Thanks, Eric On Tuesday, August 30, 2016 12:58 PM, Andrew Wang wrote: I'll put my own +1 on it: * Built from source * Started pseudo cluster and ran Pi job successfully On Tue, Aug 30, 2016 at 10:17 AM, Zhe Zhang wrote: > > Thanks Andrew for the great work! It's really exciting to finally see a > Hadoop 3 RC. > > I noticed CHANGES and RELEASENOTES markdown files which were not in > previous RCs like 2.7.3. What are good tools to verify them? I tried > reading them on IntelliJ but format looks odd. > > I'm still testing the RC: > - Downloaded and verified checksum > - Built from source > - Will start small cluster and test simple programs, focusing on EC > functionalities > > -- Zhe > > On Tue, Aug 30, 2016 at 8:51 AM Andrew Wang > wrote: > >> Hi all, >> >> Thanks to the combined work of many, many contributors, here's an RC0 for >> 3.0.0-alpha1: >> >> http://home.apache.org/~wang/3.0.0-alpha1-RC0/ >> >> alpha1 is the first in a series of planned alpha releases leading up to >> GA. >> The objective is to get an artifact out to downstreams for testing and to >> iterate quickly based on the
Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0
All on OSX 10.11.6: Verified the hashes. However, Andrew, I don't know where to find your public key, so I wasn't able to verify that they were signed by you.Built from sourceDeployed a pseudo-distributed clusterRan a few sample jobsPoked around the RM UIPoked around the attached website locally via the tarball I did find one odd thing, though. It could be a misconfiguration on my system, but I've never had this problem before with other releases (though I deal almost exclusively in 2.x and so I imagine things might be different). When I run a sleep job, I do not see any diagnostics/logs/counters printed out by the client. Initially I ran the job like I would on 2.7 and it failed (because I had not set yarn.app.mapreduce.am.env and mapreduce.admin.user.env), but I didn't see anything until I looked at the RM UI. There I was able to see all of the logs for the failed job and diagnose the issue. Then, once I fixed my parameters and ran the job again, I still didn't see any diagnostics/logs/counters. ebadger@foo: env | grep HADOOPHADOOP_HOME=/Users/ebadger/Downloads/hadoop-3.0.0-alpha1-src/hadoop-dist/target/hadoop-3.0.0-alpha1/HADOOP_CONF_DIR=/Users/ebadger/confebadger@foo: $HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-alpha1-tests.jar sleep -Dyarn.app.mapreduce.am.env="HADOOP_MAPRED_HOME=$HADOOP_HOME" -Dmapreduce.admin.user.env="HADOOP_MAPRED_HOME=$HADOOP_HOME" -mt 1 -rt 1 -m 1 -r 1WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.ebadger@foo: After running the above command, the RM UI showed a successful job, but as you can see, I did not have anything printed onto the command line. Hopefully this is just a misconfiguration on my part, but I figured that I would point it out just in case. Thanks, Eric On Tuesday, August 30, 2016 12:58 PM, Andrew Wang wrote: I'll put my own +1 on it: * Built from source * Started pseudo cluster and ran Pi job successfully On Tue, Aug 30, 2016 at 10:17 AM, Zhe Zhang wrote: > > Thanks Andrew for the great work! It's really exciting to finally see a > Hadoop 3 RC. > > I noticed CHANGES and RELEASENOTES markdown files which were not in > previous RCs like 2.7.3. What are good tools to verify them? I tried > reading them on IntelliJ but format looks odd. > > I'm still testing the RC: > - Downloaded and verified checksum > - Built from source > - Will start small cluster and test simple programs, focusing on EC > functionalities > > -- Zhe > > On Tue, Aug 30, 2016 at 8:51 AM Andrew Wang > wrote: > >> Hi all, >> >> Thanks to the combined work of many, many contributors, here's an RC0 for >> 3.0.0-alpha1: >> >> http://home.apache.org/~wang/3.0.0-alpha1-RC0/ >> >> alpha1 is the first in a series of planned alpha releases leading up to >> GA. >> The objective is to get an artifact out to downstreams for testing and to >> iterate quickly based on their feedback. So, please keep that in mind when >> voting; hopefully most issues can be addressed by future alphas rather >> than >> future RCs. >> >> Sorry for getting this out on a Tuesday, but I'd still like this vote to >> run the normal 5 days, thus ending Saturday (9/3) at 9AM PDT. I'll extend >> if we lack the votes. >> >> Please try it out and let me know what you think. >> >> Best, >> Andrew >> >
Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0
> On Aug 30, 2016, at 10:17 AM, Zhe Zhang wrote: > > Thanks Andrew for the great work! It's really exciting to finally see a > Hadoop 3 RC. > > I noticed CHANGES and RELEASENOTES markdown files which were not in > previous RCs like 2.7.3. What are good tools to verify them? I tried > reading them on IntelliJ but format looks odd. The site tarball has them converted to HTML. I've also re-run the versions that I keep on my gitlab account. (Since the data comes from JIRA, the content should be the same but the format and ordering might be different since I use the master branch of Yetus.) https://gitlab.com/_a__w_/eco-release-metadata/tree/master/HADOOP/3.0.0-alpha1 It also looks like IntelliJ has a few different markdown plug-ins. You'll want one that supports what is generally referred to as MultiMarkdown or Github-Flavored Markdown (GFM) since releasedocmaker uses the table extension format found in that specification. (It's an extremely common extension so I'm sure one of them supports it.) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0
I'll put my own +1 on it: * Built from source * Started pseudo cluster and ran Pi job successfully On Tue, Aug 30, 2016 at 10:17 AM, Zhe Zhang wrote: > > Thanks Andrew for the great work! It's really exciting to finally see a > Hadoop 3 RC. > > I noticed CHANGES and RELEASENOTES markdown files which were not in > previous RCs like 2.7.3. What are good tools to verify them? I tried > reading them on IntelliJ but format looks odd. > > I'm still testing the RC: > - Downloaded and verified checksum > - Built from source > - Will start small cluster and test simple programs, focusing on EC > functionalities > > -- Zhe > > On Tue, Aug 30, 2016 at 8:51 AM Andrew Wang > wrote: > >> Hi all, >> >> Thanks to the combined work of many, many contributors, here's an RC0 for >> 3.0.0-alpha1: >> >> http://home.apache.org/~wang/3.0.0-alpha1-RC0/ >> >> alpha1 is the first in a series of planned alpha releases leading up to >> GA. >> The objective is to get an artifact out to downstreams for testing and to >> iterate quickly based on their feedback. So, please keep that in mind when >> voting; hopefully most issues can be addressed by future alphas rather >> than >> future RCs. >> >> Sorry for getting this out on a Tuesday, but I'd still like this vote to >> run the normal 5 days, thus ending Saturday (9/3) at 9AM PDT. I'll extend >> if we lack the votes. >> >> Please try it out and let me know what you think. >> >> Best, >> Andrew >> >
Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0
Thanks Andrew for the great work! It's really exciting to finally see a Hadoop 3 RC. I noticed CHANGES and RELEASENOTES markdown files which were not in previous RCs like 2.7.3. What are good tools to verify them? I tried reading them on IntelliJ but format looks odd. I'm still testing the RC: - Downloaded and verified checksum - Built from source - Will start small cluster and test simple programs, focusing on EC functionalities -- Zhe On Tue, Aug 30, 2016 at 8:51 AM Andrew Wang wrote: > Hi all, > > Thanks to the combined work of many, many contributors, here's an RC0 for > 3.0.0-alpha1: > > http://home.apache.org/~wang/3.0.0-alpha1-RC0/ > > alpha1 is the first in a series of planned alpha releases leading up to GA. > The objective is to get an artifact out to downstreams for testing and to > iterate quickly based on their feedback. So, please keep that in mind when > voting; hopefully most issues can be addressed by future alphas rather than > future RCs. > > Sorry for getting this out on a Tuesday, but I'd still like this vote to > run the normal 5 days, thus ending Saturday (9/3) at 9AM PDT. I'll extend > if we lack the votes. > > Please try it out and let me know what you think. > > Best, > Andrew >
[VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0
Hi all, Thanks to the combined work of many, many contributors, here's an RC0 for 3.0.0-alpha1: http://home.apache.org/~wang/3.0.0-alpha1-RC0/ alpha1 is the first in a series of planned alpha releases leading up to GA. The objective is to get an artifact out to downstreams for testing and to iterate quickly based on their feedback. So, please keep that in mind when voting; hopefully most issues can be addressed by future alphas rather than future RCs. Sorry for getting this out on a Tuesday, but I'd still like this vote to run the normal 5 days, thus ending Saturday (9/3) at 9AM PDT. I'll extend if we lack the votes. Please try it out and let me know what you think. Best, Andrew