Re: [ANNOUNCE] Eric Badger is now a committer!

[Suma Shivaprasad]

Congrats Eric !

On Thu, Mar 7, 2019 at 5:06 AM zhankun tang  wrote:

> Congratulations!
>
> Zhankun
>
> 俊平堵 于2019年3月7日 周四下午12:34写道：
>
> > Congrats, Eric!
> >
> > Thanks,
> >
> > Junping
> >
> > Eric Payne  于2019年3月6日周三 上午1:20写道：
> >
> > > It is my pleasure to announce that Eric Badger has accepted an
> invitation
> > > to become a Hadoop Core committer.
> > >
> > > Congratulations, Eric! This is well-deserved!
> > >
> > > -Eric Payne
> > >
> >
>

Re: [VOTE] Release Apache Hadoop 3.1.2 - RC1

[Suma Shivaprasad]

+1

- Built from source
- Deployed a single node cluster
- Ran few MR and Distributed Shell Jobs
- Check YARN UI2 flow with Yarn services and jobs

Thanks
Suma

On Mon, Feb 4, 2019 at 7:51 PM Wilfred Spiegelenburg
 wrote:

> +1 (non binding)
>
> - build from source on MacOSX 10.14.2, 1.8.0u181
> - successful native build on Ubuntu 16.04.3
> - deployed a single node cluster  (openjdk 1.8u191 / centos 7.5)
> - uploaded the MR framework
> - configured YARN with the FS
> - ran multiple MR jobs
>
> > On 29 Jan 2019, at 17:19, Sunil G  wrote:
> >
> > Hi Folks,
> >
> > On behalf of Wangda, we have an RC1 for Apache Hadoop 3.1.2.
> >
> > The artifacts are available here:
> > http://home.apache.org/~sunilg/hadoop-3.1.2-RC1/
> >
> > The RC tag in git is release-3.1.2-RC1:
> > https://github.com/apache/hadoop/commits/release-3.1.2-RC1
> >
> > The maven artifacts are available via repository.apache.org at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1215
> >
> > This vote will run 5 days from now.
> >
> > 3.1.2 contains 325 [1] fixed JIRA issues since 3.1.1.
> >
> > We have done testing with a pseudo cluster and distributed shell job.
> >
> > My +1 to start.
> >
> > Best,
> > Wangda Tan and Sunil Govindan
> >
> > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.2)
> > ORDER BY priority DESC
>
>
> Wilfred Spiegelenburg | Software Engineer
> cloudera.com 
>
>
>
>
>
>
>
>

Re: [VOTE] Propose to start new Hadoop sub project "submarine"

[Suma Shivaprasad]

+1 ( non binding)

On Mon, Feb 4, 2019 at 7:23 AM Shashikant Banerjee <
sbaner...@hortonworks.com> wrote:

> +1 (non-binding)
>
> Thanks
> Shashi
>
> On 2/4/19, 8:27 PM, "Elek, Marton"  wrote:
>
> +1 (non-binding)
>
> (my arguments are in the discuss thread. small move, huge benefit)
>
> Thanks,
> Marton
>
> On 2/1/19 11:15 PM, Wangda Tan wrote:
> > Hi all,
> >
> > According to positive feedbacks from the thread [1]
> >
> > This is vote thread to start a new subproject named
> "hadoop-submarine"
> > which follows the release process already established for ozone.
> >
> > The vote runs for usual 7 days, which ends at Feb 8th 5 PM PDT.
> >
> > Thanks,
> > Wangda Tan
> >
> > [1]
> >
> https://lists.apache.org/thread.html/f864461eb188bd12859d51b0098ec38942c4429aae7e4d001a633d96@%3Cyarn-dev.hadoop.apache.org%3E
> >
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>
>
>

Re: [DISCUSS] Making submarine to different release model like Ozone

[Suma Shivaprasad]

+1. Thanks for bringing this up Wangda.

Makes sense to have Submarine follow its own release cadence given the good
momentum/adoption so far. Also, making it run with older versions of Hadoop
would drive higher adoption.

Suma

On Fri, Feb 1, 2019 at 9:40 AM Eric Yang  wrote:

> Submarine is an application built for YARN framework, but it does not have
> strong dependency on YARN development.  For this kind of projects, it would
> be best to enter Apache Incubator cycles to create a new community.  Apache
> commons is the only project other than Incubator that has independent
> release cycles.  The collection is large, and the project goal is
> ambitious.  No one really knows which component works with each other in
> Apache commons.  Hadoop is a much more focused project on distributed
> computing framework and not incubation sandbox.  For alignment with Hadoop
> goals, and we want to prevent Hadoop project to be overloaded while
> allowing good ideas to be carried forwarded in Apache incubator.  Put on my
> Apache Member hat, my vote is -1 to allow more independent subproject
> release cycle in Hadoop project that does not align with Hadoop project
> goals.
>
> Apache incubator process is highly recommended for Submarine:
> https://incubator.apache.org/policy/process.html This allows Submarine to
> develop for older version of Hadoop like Spark works with multiple versions
> of Hadoop.
>
> Regards,
> Eric
>
> On 1/31/19, 10:51 PM, "Weiwei Yang"  wrote:
>
> Thanks for proposing this Wangda, my +1 as well.
> It is amazing to see the progress made in Submarine last year, the
> community grows fast and quiet collaborative. I can see the reasons to get
> it release faster in its own cycle. And at the same time, the Ozone way
> works very well.
>
> —
> Weiwei
> On Feb 1, 2019, 10:49 AM +0800, Xun Liu , wrote:
> > +1
> >
> > Hello everyone,
> >
> > I am Xun Liu, the head of the machine learning team at Netease
> Research Institute. I quite agree with Wangda.
> >
> > Our team is very grateful for getting Submarine machine learning
> engine from the community.
> > We are heavy users of Submarine.
> > Because Submarine fits into the direction of our big data team's
> hadoop technology stack,
> > It avoids the needs to increase the manpower investment in learning
> other container scheduling systems.
> > The important thing is that we can use a common YARN cluster to run
> machine learning,
> > which makes the utilization of server resources more efficient, and
> reserves a lot of human and material resources in our previous years.
> >
> > Our team have finished the test and deployment of the Submarine and
> will provide the service to our e-commerce department (
> http://www.kaola.com/) shortly.
> >
> > We also plan to provides the Submarine engine in our existing YARN
> cluster in the next six months.
> > Because we have a lot of product departments need to use machine
> learning services,
> > for example:
> > 1) Game department (http://game.163.com/) needs AI battle training,
> > 2) News department (http://www.163.com) needs news recommendation,
> > 3) Mailbox department (http://www.163.com) requires anti-spam and
> illegal detection,
> > 4) Music department (https://music.163.com/) requires music
> recommendation,
> > 5) Education department (http://www.youdao.com) requires voice
> recognition,
> > 6) Massive Open Online Courses (https://open.163.com/) requires
> multilingual translation and so on.
> >
> > If Submarine can be released independently like Ozone, it will help
> us quickly get the latest features and improvements, and it will be great
> helpful to our team and users.
> >
> > Thanks hadoop Community!
> >
> >
> > > 在 2019年2月1日，上午2:53，Wangda Tan  写道：
> > >
> > > Hi devs,
> > >
> > > Since we started submarine-related effort last year, we received a
> lot of
> > > feedbacks, several companies (such as Netease, China Mobile, etc.)
> are
> > > trying to deploy Submarine to their Hadoop cluster along with big
> data
> > > workloads. Linkedin also has big interests to contribute a
> Submarine TonY (
> > > https://github.com/linkedin/TonY) runtime to allow users to use
> the same
> > > interface.
> > >
> > > From what I can see, there're several issues of putting Submarine
> under
> > > yarn-applications directory and have same release cycle with
> Hadoop:
> > >
> > > 1) We started 3.2.0 release at Sep 2018, but the release is done
> at Jan
> > > 2019. Because of non-predictable blockers and security issues, it
> got
> > > delayed a lot. We need to iterate submarine fast at this point.
> > >
> > > 2) We also see a lot of requirements to use Submarine on older
> Hadoop
> > > releases such as 2.x. Many companies may not upgrade Hadoop to 3.x
> in a
> > > short time, but the requirement t

Re: [VOTE] Release Apache Hadoop 3.1.1 - RC0

[Suma Shivaprasad]

Thanks Wangda!

+1 (non-binding)

Tested the following:
  - Built from source
  - Setup single node cluster
  - Tested Dynamic queues
  - Tested MR and DS with default, docker runtime
  - Tested Yarn Services with various restart policies

Thanks
Suma


On Tue, Aug 7, 2018 at 2:45 PM Eric Payne 
wrote:

> Thanks Wangda for creating this release.
>
> +1 (binding)
> Tested:
> - Built from source
> - Deployed to 6-node, multi-tennant, unsecured pseudo cluster with
> hierarchical queue structure (CS)
> - Refreshed queue (CS) properties
> - Intra-queue preemption (CS)
> - inter-queue preemption (CS)
> - User weights (CS)
>
> Issues:
> - Inter-queue preemption seems to be preempting unnecessarily (flapping)
> when the queue balancing feature is enabled. This does not seem to be
> specific to this release.
> - The preemption-to-balance-queue-after-satisfied.enabled property seems
> to always be enabled, but again, that is not specific to this release.
>
>
> Eric
>
>
> On Thursday, August 2, 2018, 1:44:22 PM CDT, Wangda Tan <
> wheele...@gmail.com> wrote:
>
>
>
>
>
> Hi folks,
>
> I've created RC0 for Apache Hadoop 3.1.1. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.1-RC0/
>
> The RC tag in git is release-3.1.1-RC0:
> https://github.com/apache/hadoop/commits/release-3.1.1-RC0
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1139/
>
> You can find my public key at
> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>
> This vote will run 5 days from now.
>
> 3.1.1 contains 435 [1] fixed JIRA issues since 3.1.0.
>
> I have done testing with a pseudo cluster and distributed shell job. My +1
> to start.
>
> Best,
> Wangda Tan
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.1)
> ORDER BY priority DESC
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Suma Shivaprasad]

+1 (non binding)


*Verified - User Group Queue mapping - Node labels with New UI- Dynamic
queuesThanksSuma*


On Wed, Apr 4, 2018 at 11:48 AM, Steve Loughran 
wrote:

> that's "dangerously interesting". I think you are right, and I also think
> it'll just be the version files which get generated
>
> anyway, +1 binding
>
> * ran my new Hadoop-3 profile on spark (SPARK-23807), with the committer
> binding, then my downstream tests. All is well, provided you also have a
> spark hive JAR patched to accept hadoop 3 as a legitimate hadoop version.
> That's an ongoing issue in the Spark project. With that JAR on my CP my
> downstream tests were all happy (yesterday)
>
> * today the staging files seem to be missing, at least maven is unable to
> find them even when I turn the spark snapshots-and-staging profile on.
> That'll be the maven dist process at play, nothing else
>
> On 4 Apr 2018, at 04:13, Wangda Tan mailto:wh
> eele...@gmail.com>> wrote:
>
> Hi Vinod / Arpit,
>
> I checked following versions:
> - 2.6.5 / 2.7.5 / 2.8.3 / 2.9.0 / 3.0.1:
>
> Jars in maven repo [1] are *always* different from jars in the binary
> tarball [2]: (I only checked hadoop-yarn-api-version.jar)
>
> (Following numbers are sizes of the jar)
> 2.6.5:
> - Jar in Maven: 1896185
> - Jar in tarball: 1891485
>
> 2.7.5:
> - Jar in Maven: 2039371 (md5: 15e76f7c734b49315ef2bce952509ddf)
> - Jar in tarball: 2039371 (md5: 0ef9f42f587401f5b49b39f27459f3ef)
> (Even size is same, md5 is different)
>
> 2.8.3:
> - Jar in Maven: 2451433
> - Jar in tarball: 2438975
>
> 2.9.0:
> - Jar in Maven: 2791477
> - Jar in tarball: 289
>
> 3.0.1:
> - Jar in Maven: 2852604
> - Jar in tarball: 2851373
>
> I guess the differences come from our release process.
>
> Thanks,
> Wangda
>
> [1] Maven jars are downloaded from
> https://repository.apache.org/service/local/repositories/
> releases/content/org/apache/hadoop/hadoop-yarn-api/
> /hadoop-yarn-api-.jar
> [2] Binary tarballs downloaded from http://apache.claz.org/hadoop/common/
>
>
> On Tue, Apr 3, 2018 at 4:25 PM, Vinod Kumar Vavilapalli <
> vino...@apache.org>
> wrote:
>
> We vote on the source code. The binaries are convenience artifacts.
>
> This is what I would do - (a) Just replace both the maven jars as well as
> the binaries to be consistent and correct. And then (b) Give a couple more
> days for folks who tested on the binaries to reverify - I count one such
> clear vote as of now.
>
> Thanks
> +Vinod
>
>
> On Apr 3, 2018, at 3:30 PM, Wangda Tan mailto:wh
> eele...@gmail.com>> wrote:
>
> HI Arpit,
>
> I think it won't match if we do rebuild. It should be fine as far as
> they're signed, correct? I don't see any policy doesn't allow this.
>
> Thanks,
> Wangda
>
>
> On Tue, Apr 3, 2018 at 9:33 AM, Arpit Agarwal  mailto:aagar...@hortonworks.com>>
> wrote:
>
> Thanks Wangda, I see the shaded jars now.
>
> Are the repo jars required to be the same as the binary release? They
> don’t match right now, probably they got rebuilt.
>
> +1 (binding), modulo that remaining question.
>
> * Verified signatures
> * Verified checksums for source and binary artefacts
> * Sanity checked jars on r.a.o.
> * Built from source
> * Deployed to 3 node secure cluster with NameNode HA
> * Verified HDFS web UIs
> * Tried out HDFS shell commands
> * Ran sample MapReduce jobs
>
> Thanks!
>
>
> --
> From: Wangda Tan mailto:wheele...@gmail.com>>
> Date: Monday, April 2, 2018 at 9:25 PM
> To: Arpit Agarwal mailto:aagarwal@hortonworks.
> com>>
> Cc: Gera Shegalov mailto:ger...@gmail.com>>, Sunil G <
> sun...@apache.org>, "
> yarn-...@hadoop.apache.org" <
> yarn-...@hadoop.apache.org>, Hdfs-dev <
> hdfs-dev@hadoop.apache.org>, Hadoop
> Common mailto:common-...@hadoop.apache.org
> >>,
> "mapreduce-...@hadoop.apache.org"
> mailto:mapreduce-...@hadoop.apache.org>>,
> Vinod Kumar Vavilapalli mailto:vino...@apache.org>>
> Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
>
> As pointed by Arpit, the previously deployed shared jars are incorrect.
> Just redeployed jars and staged. @Arpit, could you please check the updated
> Maven repo? https://repository.apache.org/content/repositories/
> orgapachehadoop-1092
>
> Since the jars inside binary tarballs are correct (
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't
> need roll another RC, just update Maven repo should be sufficient.
>
> Best,
> Wangda
>
>
> On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan 
> wrote:
> Hi Arpit,
>
> Thanks for pointing out this.
>
> I just removed all .md5 files from artifacts. I found md5 checksums still
> exist in .mds files and I didn't remove them from .mds file because it is
> generated by create-release script and Apache guidance is "should not"