Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Sunil Govind]

+1 (non-binding). Thanks Junping for the effort.

I have used release package and verified below cases
- Ran MR sleep job and wordcount successfully in where nodes are configured
with labels.
- Verified application priority feature and I could see high priority apps
are getting resource over lower priority apps when configured
- Verified RM web UI pages and looks fine (priority could be seen)
- Intra-queue preemption related to app priority also seems fine

Thanks
Sunil


On Fri, Mar 17, 2017 at 2:48 PM Junping Du  wrote:

> Hi all,
>  With fix of HDFS-11431 get in, I've created a new release candidate
> (RC3) for Apache Hadoop 2.8.0.
>
>  This is the next minor release to follow up 2.7.0 which has been
> released for more than 1 year. It comprises 2,900+ fixes, improvements, and
> new features. Most of these commits are released for the first time in
> branch-2.
>
>   More information about the 2.8.0 release plan can be found here:
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>
>   New RC is available at:
> http://home.apache.org/~junping_du/hadoop-2.8.0-RC3
>
>   The RC tag in git is: release-2.8.0-RC3, and the latest commit id
> is: 91f2b7a13d1e97be65db92ddabc627cc29ac0009
>
>   The maven artifacts are available via repository.apache.org at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1057
>
>   Please try the release and vote; the vote will run for the usual 5
> days, ending on 03/22/2017 PDT time.
>
> Thanks,
>
> Junping
>

Re: [jira] [Created] (HADOOP-13998) initial s3guard preview

[Sunil Govind]

Mmm

On Wed, Jan 18, 2017, 7:27 PM Steve Loughran (JIRA)  wrote:

> Steve Loughran created HADOOP-13998:
> ---
>
>  Summary: initial s3guard preview
>  Key: HADOOP-13998
>  URL: https://issues.apache.org/jira/browse/HADOOP-13998
>  Project: Hadoop Common
>   Issue Type: Sub-task
> Reporter: Steve Loughran
>
>
> JIRA to link in all the things we think are needed for a preview/merge
> into trunk
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: Jenkins patch-compile fails on trunk

[Sunil Govind]

Hi

YARN-5868 is created to handle this issue.

Thanks
Sunil


On Fri, Nov 11, 2016 at 6:09 AM Wangda Tan  wrote:

> This looks like bug caused by npm, looking into it now.
>
> Thanks,
> Wangda
>
> On Thu, Nov 10, 2016 at 4:00 PM, Zhe Zhang  wrote:
>
> > https://builds.apache.org/job/PreCommit-HDFS-Build/17517/
> > artifact/patchprocess/patch-compile-root.txt
> >
> > Looks related to yarn-ui? The HDFS-10872 patch doesn't touch YARN, but
> the
> > patch-compile failed at YARN.
> >
> > Thanks,
> > Zhe
> >
>

Re: [VOTE] Release Apache Hadoop 2.7.3 RC2

[Sunil Govind]

Hi All

+1 non-binding

1. Built from source and deployed 2 node cluster
2. Tested node labels basic scenarios
3. Ran sleep and wordcount.
4. Verified few cli commands and web UI.

Thanks
Sunil

On Thu, Aug 18, 2016 at 7:35 AM Vinod Kumar Vavilapalli 
wrote:

> Hi all,
>
> I've created a new release candidate RC2 for Apache Hadoop 2.7.3.
>
> As discussed before, this is the next maintenance release to follow up
> 2.7.2.
>
> The RC is available for validation at:
> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/ <
> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/>
>
> The RC tag in git is: release-2.7.3-RC2
>
> The maven artifacts are available via repository.apache.org <
> http://repository.apache.org/> at
> https://repository.apache.org/content/repositories/orgapachehadoop-1046 <
> https://repository.apache.org/content/repositories/orgapachehadoop-1046>
>
> The release-notes are inside the tar-balls at location
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
> hosted this at
> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/releasenotes.html <
> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/releasenotes.html> for
> your quick perusal.
>
> As you may have noted,
>  - few issues with RC0 forced a RC1 [1]
>  - few more issues with RC1 forced a RC2 [2]
>  - a very long fix-cycle for the License & Notice issues (HADOOP-12893)
> caused 2.7.3 (along with every other Hadoop release) to slip by quite a
> bit. This release's related discussion thread is linked below: [3].
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0:
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106
> 
> [2] [VOTE] Release Apache Hadoop 2.7.3 RC1:
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg26336.html <
> https://www.mail-archive.com/hdfs-dev@hadoop.apache.org/msg26336.html>
> [3] 2.7.3 release plan:
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <
> http://markmail.org/thread/6yv2fyrs4jlepmmr>

Re: [VOTE] Release Apache Hadoop 2.7.3 RC0

[Sunil Govind]

Hi All
+1 (non-binding)
- Compiled and created tar ball from source
- Tested few MR jobs with node labels
- Verified UI

Thanks
Sunil

On Sat, Jul 23, 2016 at 7:45 AM Vinod Kumar Vavilapalli 
wrote:

> Hi all,
>
> I've created a release candidate RC0 for Apache Hadoop 2.7.3.
>
> As discussed before, this is the next maintenance release to follow up
> 2.7.2.
>
> The RC is available for validation at:
> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ <
> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>
> The RC tag in git is: release-2.7.3-RC0
>
> The maven artifacts are available via repository.apache.org <
> http://repository.apache.org/> at
> https://repository.apache.org/content/repositories/orgapachehadoop-1040/ <
> https://repository.apache.org/content/repositories/orgapachehadoop-1040/>
>
> The release-notes are inside the tar-balls at location
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
> hosted this at
> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html <
> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for
> your quick perusal.
>
> As you may have noted, a very long fix-cycle for the License & Notice
> issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release)
> to slip by quite a bit. This release's related discussion thread is linked
> below: [1].
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [1]: 2.7.3 release plan:
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <
> http://markmail.org/thread/6yv2fyrs4jlepmmr>

Re: Failure to submit job on trunk hadoop cluster

[Sunil Govind]

Hi Tsuyoshi

I think you might need to set below environment variables for
HADOOP_MAPRED_HOME

mapreduce.admin.user.env
yarn.app.mapreduce.am.env

Thanks
Sunil

On Mon, Jun 27, 2016 at 4:10 PM Tsuyoshi Ozawa  wrote:

> Hi,
>
> Have anyone tried to submit jobs on trunk cluster?
> I failed to launch jobs by class not found error with following error
> messages. Should I change environment variables to setup from
> branch-2's one? I'm using the same configuration as Hadoop 2 can run.
>
> $ hadoop jar
> ./share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-alpha1-SNAPSHOT.jar
> randomtextwriter 2
>
> -> error
>
> $ yarn logs -applicationId application_1467022691488_0004
> ...
>
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:946)
>   at
> org.apache.hadoop.util.Shell.run(Shell.java:850)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1144)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:238)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:385)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:281)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:89)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> Container exited with a non-zero exit code 1. Last 4096 bytes of
> stderr :
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
>
> Thanks,
> - Tsuyoshi
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 2.6.4 RC0

[Sunil Govind]

+1 (non-binding) with one note,

- Installed tar ball from source and brought up the cluster
- Ran few MR jobs and those are working fine.
- UI and REST apis are also looks fine. Ran few rest queries for this test.
- Tested node label feature and works fine. (one observation during this
testing)

One note to add here. "-Dmapreduce.job.node-label-expression" support is
not there 2.6.4. (MAPREDUCE-6304). So we cannot specify label to
application while submitting.  I think this also can be ported to 2.6
branch so it will help to specify labels at submission time.

Thanks and Regards
Sunil

On Wed, Feb 3, 2016 at 12:31 PM Junping Du  wrote:

> Hi community folks,
>I've created a release candidate RC0 for Apache Hadoop 2.6.4 (the next
> maintenance release to follow up 2.6.3.) according to email thread of
> release plan 2.6.4 [1]. Below is details of this release candidate:
>
> The RC is available for validation at:
> *http://people.apache.org/~junping_du/hadoop-2.6.4-RC0/
> *
>
> The RC tag in git is: release-2.6.4-RC0
>
> The maven artifacts are staged via repository.apache.org at:
> *https://repository.apache.org/content/repositories/orgapachehadoop-1028/?
>  >*
>
> You can find my public key at:
> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>
> Please try the release and vote. The vote will run for the usual 5 days.
>
> Thanks!
>
>
> Cheers,
>
> Junping
>
>
> [1]: 2.6.4 release plan: http://markmail.org/message/fk3ud3c665lscvx5?
>
>

Re: [VOTE] Release Apache Hadoop 2.7.2 RC2

[Sunil Govind]

+1 (Non Binding)

* Built tar ball from source and deployed
* Verified few MR Jobs for various nodelabel and preemption cases.
* Verified RM Web UI and REST queries. looks fine.

Thanks and Regards
Sunil G

On Fri, Jan 15, 2016 at 10:27 AM Vinod Kumar Vavilapalli 
wrote:

> Hi all,
>
> I've created an updated release candidate RC2 for Apache Hadoop 2.7.2.
>
> As discussed before, this is the next maintenance release to follow up
> 2.7.1.
>
> The RC is available for validation at:
> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/
>
> The RC tag in git is: release-2.7.2-RC2
>
> The maven artifacts are available via repository.apache.org <
> http://repository.apache.org/> at
> https://repository.apache.org/content/repositories/orgapachehadoop-1027 <
> https://repository.apache.org/content/repositories/orgapachehadoop-1027>
>
> The release-notes are inside the tar-balls at location
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
> hosted this at
> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/releasenotes.html <
> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for
> your quick perusal.
>
> As you may have noted,
>  - I terminated the RC1 related voting thread after finding out that we
> didn’t have a bunch of patches that are already in the released 2.6.3
> version. After a brief discussion, we decided to keep the parallel 2.6.x
> and 2.7.x releases incremental, see [4] for this discussion.
>  - The RC0 related voting thread got halted due to some critical issues.
> It took a while again for getting all those blockers out of the way. See
> the previous voting thread [3] for details.
>  - Before RC0, an unusually long 2.6.3 release caused 2.7.2 to slip by
> quite a bit. This release's related discussion threads are linked below:
> [1] and [2].
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [1]: 2.7.2 release plan: http://markmail.org/message/oozq3gvd4nhzsaes <
> http://markmail.org/message/oozq3gvd4nhzsaes>
> [2]: Planning Apache Hadoop 2.7.2
> http://markmail.org/message/iktqss2qdeykgpqk <
> http://markmail.org/message/iktqss2qdeykgpqk>
> [3]: [VOTE] Release Apache Hadoop 2.7.2 RC0:
> http://markmail.org/message/5txhvr2qdiqglrwc <
> http://markmail.org/message/5txhvr2qdiqglrwc>
> [4] Retracted [VOTE] Release Apache Hadoop 2.7.2 RC1:
> http://markmail.org/thread/n7ljbsnquihn3wlw

Re: [VOTE] Release Apache Hadoop 2.7.2 RC1

[Sunil Govind]

+1 (Non Binding)

* Built tar ball from source and deployed
* Ran few MR Jobs successfully. Also tested preemption, node labels sanity
scenarios including  HA.
* Verified RM Web UI and ran few REST queries. Seems fine.
* CLI commands related to refresh are also working fine.

Thanks and Regards
Sunil G


On Mon, Dec 21, 2015 at 5:41 PM Naganarasimha G R (Naga) <
garlanaganarasi...@huawei.com> wrote:

> +1 (Non binding)
>
> * Release notes and CHANGES.txt looked fine
> * Verified md5 checksums for both source and binary tar balls
> * Ran `mvn apache-rat:check` on source tarball, passed
> * Installed pseudo cluster
> * Did some basic filesystem operations
> * Ran a grep, sleepjob, word count successfully
> * Verified the jobs RM, job history server and ATS Web ui's successfully
> * Verified some RM REST apis
> * Verified basic admin operations of Node Labels.
>
> Regards,
>  +Naga
> 
> From: Akira AJISAKA [ajisa...@oss.nttdata.co.jp]
> Sent: Monday, December 21, 2015 14:29
> To: common-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org;
> hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org
> Subject: Re: [VOTE] Release Apache Hadoop 2.7.2 RC1
>
> +1 (binding)
>
> - Downloaded source tarball
> - Verified signatures and checksums
> - Compiled and built a single node cluster
> - Confirmed commons-collections is upgraded to 3.2.2
> - Compiled Hive 1.2.1 and Tez 0.7.0/0.8.1-alpha using Hadoop 2.7.2 pom
> successfully
> - Ran some Hive on Tez/MRv2 queries successfully
>
> On 12/19/15 02:24, Jason Lowe wrote:
> > +1 (binding)
> > - Verified signatures and digests- Spot checked CHANGES.txt files-
> Successfully performed a native build from source- Deployed to a single
> node cluster and ran sample jobs
> > We have been running with the fix for YARN-4354 on two of our clusters
> for some time with no issues, so I feel confident that prior blocker is now
> fixed.
> > Jason
> >
> >
> >From: Vinod Kumar Vavilapalli 
> >   To: Hadoop Common ;
> hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org;
> mapreduce-...@hadoop.apache.org
> > Cc: Vinod Kumar Vavilapalli 
> >   Sent: Wednesday, December 16, 2015 8:49 PM
> >   Subject: [VOTE] Release Apache Hadoop 2.7.2 RC1
> >
> > Hi all,
> >
> > I've created a release candidate RC1 for Apache Hadoop 2.7.2.
> >
> > As discussed before, this is the next maintenance release to follow up
> 2.7.1.
> >
> > The RC is available for validation at:
> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/ <
> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/>
> >
> > The RC tag in git is: release-2.7.2-RC1
> >
> > The maven artifacts are available via repository.apache.org <
> http://repository.apache.org/> at
> https://repository.apache.org/content/repositories/orgapachehadoop-1026/ <
> https://repository.apache.org/content/repositories/orgapachehadoop-1026/>
> >
> > The release-notes are inside the tar-balls at location
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
> hosted this at
> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html for
> quick perusal.
> >
> > As you may have noted,
> >   - The RC0 related voting thread got halted due to some critical
> issues. It took a while again for getting all those blockers out of the
> way. See the previous voting thread [3] for details.
> >   - Before RC0, an unusually long 2.6.3 release caused 2.7.2 to slip by
> quite a bit. This release's related discussion threads are linked below:
> [1] and [2].
> >
> > Please try the release and vote; the vote will run for the usual 5 days.
> >
> > Thanks,
> > Vinod
> >
> > [1]: 2.7.2 release plan: http://markmail.org/message/oozq3gvd4nhzsaes <
> http://markmail.org/message/oozq3gvd4nhzsaes>
> > [2]: Planning Apache Hadoop 2.7.2
> http://markmail.org/message/iktqss2qdeykgpqk <
> http://markmail.org/message/iktqss2qdeykgpqk>
> > [3]: [VOTE] Release Apache Hadoop 2.7.2 RC0:
> http://markmail.org/message/5txhvr2qdiqglrwc
> >
> >
> >
> >
>
>

Re: [VOTE] Release Apache Hadoop 2.7.2 RC0

[Sunil Govind]

+1 (non-binding)

- Build the tar ball from source and deployed.
- Ran few MR jobs successfully along with some basic node label and
preemption verification.
- Verified RM Web UI, AM UI and Timeline UI. All pages are looking fine.

Thanks and Regards
Sunil

On Thu, Nov 12, 2015 at 10:01 AM Vinod Kumar Vavilapalli 
wrote:

> Hi all,
>
>
> I've created a release candidate RC0 for Apache Hadoop 2.7.2.
>
>
> As discussed before, this is the next maintenance release to follow up
> 2.7.1.
>
>
> The RC is available for validation at:
>
> *http://people.apache.org/~vinodkv/hadoop-2.7.2-RC0/
>
> *
>
>
> The RC tag in git is: release-2.7.2-RC0
>
>
> The maven artifacts are available via repository.apache.org at
>
> *https://repository.apache.org/content/repositories/orgapachehadoop-1023/
>
>  >*
>
>
> As you may have noted, an unusually long 2.6.3 release caused 2.7.2 to slip
> by quite a bit. This release's related discussion threads are linked below:
> [1] and [2].
>
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
>
> Thanks,
>
> Vinod
>
>
> [1]: 2.7.2 release plan: http://markmail.org/message/oozq3gvd4nhzsaes
>
> [2]: Planning Apache Hadoop 2.7.2
> http://markmail.org/message/iktqss2qdeykgpqk
>

Re: [DISCUSS] Looking to a 2.8.0 release

[Sunil Govind]

Thank you Vinod for starting this discussion.

+1 for getting a beta/alpha version for Application Priority (YARN-1963).
Major patches are already in and MAPREDUCE-5870 (making MR also to use
app-priority) is in final review stages. This feature is also tested
in-house. With 2.8.0 alpha, I feel we can see more use case based testing
on same.

A documentation JIRA is already raised for App Priority, and I will mark it
for 2.8 so that it will be done before the RC cut.

Thank You
Sunil
On Thu, Nov 12, 2015 at 2:41 AM Vinod Vavilapalli 
wrote:

> I’ll let others comment on specific features.
>
> Regarding the 3.x vs 2.x point, as I noted before on other threads, given
> all the incompatibilities in trunk it will be ways off before users can run
> their production workloads on a 3.x release. Therefore, as I was proposing
> before, we should continue the 2.x lines, but soon get started on rolling
> out a release candidate based off trunk.
>
> Like with 2.8, I’d like to go back and prepare some notes on trunk’s
> content so we can objectively discuss about it.
>
> +Vinod