Re: [VOTE] Release Apache Hadoop 2.7.3 RC2

[Jason Lowe]

+1 (binding)
- Verified signatures and digests- Successfully built from source with native 
support- Deployed a single-node cluster- Ran some sample jobs successfully

Jason

  From: Vinod Kumar Vavilapalli 
 To: "common-...@hadoop.apache.org" ; 
hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org; 
"mapreduce-dev@hadoop.apache.org"  
Cc: Vinod Kumar Vavilapalli 
 Sent: Wednesday, August 17, 2016 9:05 PM
 Subject: [VOTE] Release Apache Hadoop 2.7.3 RC2
   
Hi all,

I've created a new release candidate RC2 for Apache Hadoop 2.7.3.

As discussed before, this is the next maintenance release to follow up 2.7.2.

The RC is available for validation at: 
http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/ 


The RC tag in git is: release-2.7.3-RC2

The maven artifacts are available via repository.apache.org 
 at 
https://repository.apache.org/content/repositories/orgapachehadoop-1046 


The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
this at http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/releasenotes.html 
 for your 
quick perusal.

As you may have noted,
 - few issues with RC0 forced a RC1 [1]
 - few more issues with RC1 forced a RC2 [2]
 - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 
2.7.3 (along with every other Hadoop release) to slip by quite a bit. This 
release's related discussion thread is linked below: [3].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 

[2] [VOTE] Release Apache Hadoop 2.7.3 RC1: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg26336.html 

[3] 2.7.3 release plan: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 


   

Re: [VOTE] Release Apache Hadoop 2.7.3 RC2

[Vinod Kumar Vavilapalli]

Too late for 2.7.3 - I want us to quickly get back to a regular cadence of 
releases.

Doesn’t look like a regression IAC. Let’s put it in the next release if needed.

Thanks
+Vinod

> On Aug 22, 2016, at 6:19 AM, Brahma Reddy Battula 
>  wrote:
> 
> Felt like HDFS-8388, should be in for 2.7.3

Re: Submit, suspend and resume a mapreduce job execution

[Eric Payne]

If the only thing you want to do is make sure the reducers don't start until 
all of the maps are done, you could set 
mapreduce.job.reduce.slowstart.completedmaps to 1.0. By default, it is 0.05. 
This property defines the fraction of the number of maps in the job which 
should be complete before reduces are scheduled for the job.

If you want to have a job that does something between the maps and the 
reducers, the TEZ framework may work for you. I'm not extremely familiar with 
TEZ, but I know that it does allow arbitrary DAGs to be defined. It would 
require some amount of work to port from MapRed to TEZ, though.

Hope that helps.
-Eric Payne



- Original Message -
From: xeon Mailinglist 
To: d...@slider.incubator.apache.org; mapreduce-dev@hadoop.apache.org
Sent: Sunday, August 21, 2016 5:50 AM
Subject: Re: Submit, suspend and resume a mapreduce job execution

I know that it is not possible to suspend and resume mapreduce job, but I
really need to find a workaround. I have looked to the ChainedJobs and to
the CapacityScheduler, but I am really clueless on what to do.

The main goal was to suspend a job when the map tasks finish and the reduce
tasks start. I know that this is not possible, so I have created to jobs.
One that execute all the map tasks (Job 1), and another job that execute
all the reduce tasks (Job 2). Since I can't start a job with just running
reduce tasks, it was necessary to add an identity mapper before running the
reducers. So in the end, I have Job 1 that just executes all map tasks, and
job 2 that executes the identity mappers and the reduce tasks. But this
really kills performance. I wish I could find a way to obtain better
performance. I have thought in doing pipe of the output of Job 1 to Job 2,
but in the end I really need to stop the execution between these 2 jobs.

I have looked to the ChainedJobs and CapacityScheduler classes to see if I
could implement a way to suspend and resume a job, but I didn't do nothing
successfully. Any idea to emulate a way to suspend a job?

Sorry to say this, but I am really desperate in finding a solution.

Thanks,


On Wed, Feb 18, 2015 at 6:53 PM, Steve Loughran 
wrote:

> Afraid not.
>
> When we suspend/resume a slider application, what we are doing is shutting
> down the entire application, releasing all its YARN resources and killing
> the "Application Master". The  MapReduce engine runs its AM for the
> duration of the job; building up lots of state in that AM as to what is
> happening. Tez runs for longer, but it can dynamically change cluster size
> based on load.
>
> "Hadoop pre-emption" is a mechanism by which your cluster can be set up so
> that higher priority workloads can cause containers of lower-priority jobs
> to get killed, "pre-empted". Maybe that could be useful.
>
> -Steve
>
>
>
> On 18 February 2015 at 17:22:57, xeonmailinglist (
> xeonmailingl...@gmail.com) wrote:
>
> Hi,
>
> I noticed that YARN does not suspend or resume a mapreduce job that it
> is executing. Then, I have found Apache Slider.
> Is it possible to submit a mapreduce job with slider, and suspend and
> resume the job while executing?
>
> Thanks,
>
>

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/141/

[Aug 21, 2016 4:16:17 PM] (varunsaxena) MAPREDUCE-6762. ControlledJob#toString 
failed with NPE when job status
[Aug 21, 2016 4:40:29 PM] (liuml07) HADOOP-13527. Add Spark to CallerContext 
LimitedPrivate scope.




-1 overall


The following subsystems voted -1:
asflicense unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.security.TestDelegationToken 
   hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices 
   hadoop.yarn.server.resourcemanager.TestNodeBlacklistingOnAMFailures 
   hadoop.yarn.server.resourcemanager.TestRMAdminService 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.yarn.client.api.impl.TestAMRMClient 
   hadoop.yarn.client.api.impl.TestYarnClient 

Timed out junit tests :

   org.apache.hadoop.hdfs.TestLeaseRecovery2 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/141/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/141/artifact/out/diff-compile-javac-root.txt
  [172K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/141/artifact/out/diff-checkstyle-root.txt
  [16M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/141/artifact/out/diff-patch-pylint.txt
  [16K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/141/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/141/artifact/out/diff-patch-shelldocs.txt
  [16K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/141/artifact/out/whitespace-eol.txt
  [12M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/141/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/141/artifact/out/diff-javadoc-javadoc-root.txt
  [2.2M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/141/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [144K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/141/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/141/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [56K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/141/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [268K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/141/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/141/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt
  [124K]

   asflicense:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/141/artifact/out/patch-asflicense-problems.txt
  [4.0K]

Powered by Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org



-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

RE: [VOTE] Release Apache Hadoop 2.7.3 RC2

[Brahma Reddy Battula]

Thanks Vinod for creating new RC for 2.7.3 release.

+1 (non-binding)

- Downloaded source and Compiled, built source code

- Deployed a 3 node HA cluster

-Ran MR jobs, like: Slive,pi, teragen/terasort, etc.

- Did the regression test for handled defects.

- Verified basic CLIs for yarn,hdfs.


Felt like HDFS-8388, should be in for 2.7.3

--Brahma Reddy Battula

-Original Message-
From: Vinod Kumar Vavilapalli [mailto:vino...@apache.org] 
Sent: 18 August 2016 10:06
To: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Cc: Vinod Kumar Vavilapalli
Subject: [VOTE] Release Apache Hadoop 2.7.3 RC2

Hi all,

I've created a new release candidate RC2 for Apache Hadoop 2.7.3.

As discussed before, this is the next maintenance release to follow up 2.7.2.

The RC is available for validation at: 
http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/ 


The RC tag in git is: release-2.7.3-RC2

The maven artifacts are available via repository.apache.org 
 at 
https://repository.apache.org/content/repositories/orgapachehadoop-1046 


The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
this at http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/releasenotes.html 
 for your 
quick perusal.

As you may have noted,
 - few issues with RC0 forced a RC1 [1]
 - few more issues with RC1 forced a RC2 [2]
 - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 
2.7.3 (along with every other Hadoop release) to slip by quite a bit. This 
release's related discussion thread is linked below: [3].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 

[2] [VOTE] Release Apache Hadoop 2.7.3 RC1: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg26336.html 

[3] 2.7.3 release plan: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org