from:"Mukul Kumar Singh"

Re: [DISCUSS] Shade guava into hadoop-thirdparty

2020-04-06 Thread Mukul Kumar Singh

On 07/04/20 7:05 am, Zhankun Tang wrote:

Thanks, Wei-Chiu for the proposal. +1.

On Mon, 6 Apr 2020 at 20:17, Ayush Saxena wrote:

-Ayush

On 05-Apr-2020, at 12:43 AM, Wei-Chiu Chuang wrote:

Hi Hadoop devs,

I spent a good part of the past 7 months working with a dozen of

colleagues

to update the guava version in Cloudera's software (that includes Hadoop,
HBase, Spark, Hive, Cloudera Manager ... more than 20+ projects)

After 7 months, I finally came to a conclusion: Update to Hadoop 3.3 /
3.2.1 / 3.1.3, even if you just go from Hadoop 3.0/ 3.1.0 is going to be
really hard because of guava. Because of Guava, the amount of work to
certify a minor release update is almost equivalent to a major release
update.

That is because:
(1) Going from guava 11 to guava 27 is a big jump. There are several
incompatible API changes in many places. Too bad the Google developers

are

not sympathetic about its users.
(2) guava is used in all Hadoop jars. Not just Hadoop servers but also
client jars and Hadoop common libs.
(3) The Hadoop library is used in practically all software at Cloudera.

Here is my proposal:
(1) shade guava into hadoop-thirdparty, relocate the classpath to
org.hadoop.thirdparty.com.google.common.*
(2) make a hadoop-thirdparty 1.1.0 release.
(3) update existing references to guava to the relocated path. There are
more than 2k imports that need an update.
(4) release Hadoop 3.3.1 / 3.2.2 that contains this change.

In this way, we will be able to update guava in Hadoop in the future
without disrupting Hadoop applications.

Note: HBase already did this and this guava update project would have

been

much more difficult if HBase didn't do so.

Thoughts? Other options include
(1) force downstream applications to migrate to Hadoop client artifacts

listed here

https://hadoop.apache.org/docs/r3.1.1/hadoop-project-dist/hadoop-common/DownstreamDev.html

but
that's nearly impossible.
(2) Migrate Guava to Java APIs. I suppose this is a big project and I

can't

estimate how much work it's going to be.

Weichiu

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Force "squash and merge" option for PR merge on github UI

2019-07-17 Thread Mukul Kumar Singh

+1, Lets have Squash and merge as the default & only option on github UI.

Thanks,
Mukul

On 7/17/19 11:37 AM, Elek, Marton wrote:

Hi,

Github UI (ui!) helps to merge Pull Requests to the proposed branch.
There are three different ways to do it [1]:

1. Keep all the different commits from the PR branch and create one
additional merge commit ("Create a merge commit")

2. Squash all the commits and commit the change as one patch ("Squash
and merge")

3. Keep all the different commits from the PR branch but rebase, merge
commit will be missing ("Rebase and merge")

As only the option 2 is compatible with the existing development
practices of Hadoop (1 issue = 1 patch = 1 commit), I call for a lazy
consensus vote: If no objections withing 3 days, I will ask INFRA to
disable the options 1 and 3 to make the process less error prone.

Please let me know, what do you think,

Thanks a lot
Marton

ps: Personally I prefer to merge from local as it enables to sign the
commits and do a final build before push. But this is a different story,
this proposal is only about removing the options which are obviously
risky...

ps2: You can always do any kind of merge / commits from CLI, for example
to merge a feature branch together with keeping the history.

[1]:
https://help.github.com/en/articles/merging-a-pull-request#merging-a-pull-request-on-github

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] Move to gitbox

2018-12-11 Thread Mukul Kumar Singh

+1
-Mukul

On 12/11/18, 9:56 AM, "Weiwei Yang"  wrote:

+1

On Tue, Dec 11, 2018 at 10:51 AM Anu Engineer 
wrote:

> +1
> --Anu
>
>
> On 12/10/18, 6:38 PM, "Vinayakumar B"  wrote:
>
> +1
>
> -Vinay
>
> On Mon, 10 Dec 2018, 1:22 pm Elek, Marton 
> >
> > Thanks Akira,
> >
> > +1 (non-binding)
> >
> > I think it's better to do it now at a planned date.
> >
> > If I understood well the only bigger task here is to update all the
> > jenkins jobs. (I am happy to help/contribute what I can do)
> >
> >
> > Marton
> >
> > On 12/8/18 6:25 AM, Akira Ajisaka wrote:
> > > Hi all,
> > >
> > > Apache Hadoop git repository is in git-wip-us server and it will 
be
> > > decommissioned.
> > > If there are no objection, I'll file a JIRA ticket with INFRA to
> > > migrate to https://gitbox.apache.org/ and update documentation.
> > >
> > > According to ASF infra team, the timeframe is as follows:
> > >
> > >> - December 9th 2018 -> January 9th 2019: Voluntary (coordinated)
> > relocation
> > >> - January 9th -> February 6th: Mandated (coordinated) relocation
> > >> - February 7th: All remaining repositories are mass migrated.
> > >> This timeline may change to accommodate various scenarios.
> > >
> > > If we got consensus by January 9th, I can file a ticket with INFRA
> and
> > > migrate it.
> > > Even if we cannot got consensus, the repository will be migrated 
by
> > > February 7th.
> > >
> > > Regards,
> > > Akira
> > >
> > >
> -
> > > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> > >
> >
> > 
-
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
> >
>
>
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>


-- 
Weiwei Yang



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop Ozone 0.3.0-alpha (RC1)

2018-11-19 Thread Mukul Kumar Singh

+1.
Thanks for putting the release Marton, 

- Verified signatures and checksum
- Built from source
- Deployed Ozone on a physical cluster and ran Freon.


On 11/19/18, 7:27 PM, "Lokesh Jain"  wrote:

+1 (non-binding)

- Verified signatures and checksum
- Built from source
- Ran smoke tests

Thanks Marton for putting the release together!

- Lokesh
> On 16-Nov-2018, at 6:45 PM, Shashikant Banerjee 
 wrote:
> 
> +1 (non-binding).
> 
>  - Verified signatures
>  - Verified checksums
>  - Checked LICENSE/NOTICE files
>  - Built from source
>  - Ran smoke tests.
> 
> Thanks Marton for putting up the release together.
> 
> Thanks
> Shashi
> 
> On 11/14/18, 10:44 PM, "Elek, Marton"  wrote:
> 
>Hi all,
> 
>I've created the second release candidate (RC1) for Apache Hadoop Ozone
>0.3.0-alpha including one more fix on top of the previous RC0 
(HDDS-854)
> 
>This is the second release of Apache Hadoop Ozone. Notable changes 
since
>the first release:
> 
>* A new S3 compatible rest server is added. Ozone can be used from any
>S3 compatible tools (HDDS-434)
>* Ozone Hadoop file system URL prefix is renamed from o3:// to o3fs://
>(HDDS-651)
>* Extensive testing and stability improvements of OzoneFs.
>* Spark, YARN and Hive support and stability improvements.
>* Improved Pipeline handling and recovery.
>* Separated/dedicated classpath definitions for all the Ozone
>components. (HDDS-447)
> 
>The RC artifacts are available from:
>https://home.apache.org/~elek/ozone-0.3.0-alpha-rc1/
> 
>The RC tag in git is: ozone-0.3.0-alpha-RC1 (ebbf459e6a6)
> 
>Please try it out, vote, or just give us feedback.
> 
>The vote will run for 5 days, ending on November 19, 2018 18:00 UTC.
> 
> 
>Thank you very much,
>Marton
> 
>
PS:
> 
>The easiest way to try it out is:
> 
>1. Download the binary artifact
>2. Read the docs from ./docs/index.html
>3. TLDR; cd compose/ozone && docker-compose up -d
>4. open localhost:9874 or localhost:9876
> 
> 
> 
>The easiest way to try it out from the source:
> 
>1. mvn  install -DskipTests -Pdist -Dmaven.javadoc.skip=true -Phdds
>-DskipShade -am -pl :hadoop-ozone-dist
>2. cd hadoop-ozone/dist/target/ozone-0.3.0-alpha && docker-compose up 
-d
> 
> 
> 
>The easiest way to test basic functionality (with acceptance tests):
> 
>1. mvn  install -DskipTests -Pdist -Dmaven.javadoc.skip=true -Phdds
>-DskipShade -am -pl :hadoop-ozone-dist
>2. cd hadoop-ozone/dist/target/ozone-0.3.0-alpha/smoketest
>3. ./test.sh
> 
>-
>To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> 
> 
> 
> 
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> 


-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org




-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop Ozone 0.2.1-alpha (RC0)

2018-09-26 Thread Mukul Kumar Singh

Thanks for working on this Marton.

+1 (non-binding)
1.  Verified Checksum
2. Verified the OM and SCM UI
3. Verified freon on a single node docker based cluster.
4. Build from source

-Mukul



On 9/26/18, 6:02 PM, "Nandakumar Vadivelu"  wrote:

Thanks Marton for all effort to make this release happen.

+1 (non-binding)

1. Verified Signature and Checksum
2. Build from Sources
3. Ran 5 node cluster using docker
4. Installed pseudo distributed cluster
5. Ran RPC and REST commands against the cluster
6. Verified SCM and OM UI

-Nanda

On 9/20/18, 3:19 AM, "Elek, Marton"  wrote:

Hi all,

After the recent discussion about the first Ozone release I've created 
the first release candidate (RC0) for Apache Hadoop Ozone 0.2.1-alpha.

This release is alpha quality: it’s not recommended to use in 
production 
but we believe that it’s stable enough to try it out the feature set 
and 
collect feedback.

The RC artifacts are available from: 
https://home.apache.org/~elek/ozone-0.2.1-alpha-rc0/

The RC tag in git is: ozone-0.2.1-alpha-RC0 (968082ffa5d)

Please try the release and vote; the vote will run for the usual 5 
working days, ending on September 26, 2018 10pm UTC time.

The easiest way to try it out is:

1. Download the binary artifact
2. Read the docs at ./docs/index.html
3. TLDR; cd compose/ozone && docker-compose up -d


Please try it out, vote, or just give us feedback.

Thank you very much,
Marton

ps: At next week, we will have a BoF session at ApacheCon North Europe, 
Montreal on Monday evening. Please join, if you are interested, or need 
support to try out the package or just have any feedback.


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org




-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: HADOOP-14163 proposal for new hadoop.apache.org

2018-09-02 Thread Mukul Kumar Singh

+1, Thanks for working on this Marton.

-Mukul

On 03/09/18, 9:02 AM, "John Zhuge"  wrote:

+1 Like the new site.

On Sun, Sep 2, 2018 at 7:02 PM Weiwei Yang  wrote:

> That's really nice, +1.
>
> --
> Weiwei
>
> On Sat, Sep 1, 2018 at 4:36 AM Wangda Tan  wrote:
>
> > +1, thanks for working on this, Marton!
> >
> > Best,
> > Wangda
> >
> > On Fri, Aug 31, 2018 at 11:24 AM Arpit Agarwal  >
> > wrote:
> >
> > > +1
> > >
> > > Thanks for initiating this Marton.
> > >
> > >
> > > On 8/31/18, 1:07 AM, "Elek, Marton"  wrote:
> > >
> > > Bumping this thread at last time.
> > >
> > > I have the following proposal:
> > >
> > > 1. I will request a new git repository hadoop-site.git and import
> the
> > > new site to there (which has exactly the same content as the
> existing
> > > site).
> > >
> > > 2. I will ask infra to use the new repository as the source of
> > > hadoop.apache.org
> > >
> > > 3. I will sync manually all of the changes in the next two months
> > back
> > > to the svn site from the git (release announcements, new
> committers)
> > >
> > > IN CASE OF ANY PROBLEM we can switch back to the svn without any
> > > problem.
> > >
> > > If no-one objects within three days, I'll assume lazy consensus 
and
> > > start with this plan. Please comment if you have objections.
> > >
> > > Again: it allows immediate fallback at any time as svn repo will 
be
> > > kept
> > > as is (+ I will keep it up-to-date in the next 2 months)
> > >
> > > Thanks,
> > > Marton
> > >
> > >
> > > On 06/21/2018 09:00 PM, Elek, Marton wrote:
> > > >
> > > > Thank you very much to bump up this thread.
> > > >
> > > >
> > > > About [2]: (Just for the clarification) the content of the
> proposed
> > > > website is exactly the same as the old one.
> > > >
> > > > About [1]. I believe that the "mvn site" is perfect for the
> > > > documentation but for website creation there are more simple and
> > > > powerful tools.
> > > >
> > > > Hugo has more simple compared to jekyll. Just one binary, 
without
> > > > dependencies, works everywhere (mac, linux, windows)
> > > >
> > > > Hugo has much more powerful compared to "mvn site". Easier to
> > > create/use
> > > > more modern layout/theme, and easier to handle the content (for
> > > example
> > > > new release announcements could be generated as part of the
> release
> > > > process)
> > > >
> > > > I think it's very low risk to try out a new approach for the 
site
> > > (and
> > > > easy to rollback in case of problems)
> > > >
> > > > Marton
> > > >
> > > > ps: I just updated the patch/preview site with the recent
> releases:
> > > >
> > > > ***
> > > > * http://hadoop.anzix.net *
> > > > ***
> > > >
> > > > On 06/21/2018 01:27 AM, Vinod Kumar Vavilapalli wrote:
> > > >> Got pinged about this offline.
> > > >>
> > > >> Thanks for keeping at it, Marton!
> > > >>
> > > >> I think there are two road-blocks here
> > > >>   (1) Is the mechanism using which the website is built good
> > enough
> > > -
> > > >> mvn-site / hugo etc?
> > > >>   (2) Is the new website good enough?
> > > >>
> > > >> For (1), I just think we need more committer attention and get
> > > >> feedback rapidly and get it in.
> > > >>
> > > >> For (2), how about we do it in a different way in the interest
> of
> > > >> progress?
> > > >>   - We create a hadoop.apache.org/new-site/ where this new site
> > > goes.
> > > >>   - We then modify the existing web-site to say that there is a
> > new
> > > >> site/experience that folks can click on a link and navigate to
> > > >>   - As this new website matures and gets feedback & fixes, we
> > > finally
> > > >> pull the plug at a later point of time when we think we are 
good
> > to
> > > go.
> > > >>
> > > >> Thoughts?
> > > >>
> > > >> +Vinod
> > > >>
> > > >>> On Feb 16, 2018, at 3:10 AM, Elek, Marton 
> > wrote:
> > > >>>
> > > >>> Hi,
> > > >>>
> > > >>> I would like to bump this thread up.
> > > >>>
> > > >>> TLDR; There is a proposed version of a new hadoop site which 
is
> > > >>> available from here:
> > https://elek.github.io/hadoop-site-proposal/
> > > and
> > >

[jira] [Created] (YARN-8487) Remove the unused Variable in TestBroadcastAMRMProxyFederationPolicy#testNotifyOfResponseFromUnknownSubCluster

2018-07-02 Thread Mukul Kumar Singh (JIRA)

Mukul Kumar Singh created YARN-8487:
---

 Summary: Remove the unused Variable in 
TestBroadcastAMRMProxyFederationPolicy#testNotifyOfResponseFromUnknownSubCluster
 Key: YARN-8487
 URL: https://issues.apache.org/jira/browse/YARN-8487
 Project: Hadoop YARN
  Issue Type: Bug
  Components: amrmproxy
Reporter: Mukul Kumar Singh






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

2017-11-18 Thread Mukul Kumar Singh

Thanks Andrew for setting up this release.

+1 (non-binding)

I built from source on Mac OS X 10.13.1 Java 1.8.0_111

- Deployed on a single node cluster.
- Deployed a ViewFS cluster with two hdfs mount points.
- Performed basic sanity checks.
- Performed DFS operations(put, ls, mkdir, touch)

Thanks,
Mukul


> On 15-Nov-2017, at 3:04 AM, Andrew Wang  wrote:
> 
> Hi folks,
> 
> Thanks as always to the many, many contributors who helped with this
> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> available here:
> 
> http://people.apache.org/~wang/3.0.0-RC0/
> 
> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
> 
> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> additions include the merge of YARN resource types, API-based configuration
> of the CapacityScheduler, and HDFS router-based federation.
> 
> I've done my traditional testing with a pseudo cluster and a Pi job. My +1
> to start.
> 
> Best,
> Andrew


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC3)

2017-11-15 Thread Mukul Kumar Singh

+1 (non-binding)

I built from source on Mac OS X 10.13.1 Java 1.8.0_111

- Deployed on a single node cluster.
- Deployed a ViewFS cluster with two hdfs mount points.
- Performed basic sanity checks.
- Performed DFS operations(put, ls, mkdir, touch)

Thanks,
Mukul


> On 14-Nov-2017, at 5:40 AM, Arun Suresh  wrote:
> 
> Hi Folks,
> 
> Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will be the
> starting release for Apache Hadoop 2.9.x line - it includes 30 New Features
> with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues since
> 2.8.2.
> 
> More information about the 2.9.0 release plan can be found here:
> *https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
> *
> 
> New RC is available at: *https://home.apache.org/~asuresh/hadoop-2.9.0-RC3/
> *
> 
> The RC tag in git is: release-2.9.0-RC3, and the latest commit id is:
> 756ebc8394e473ac25feac05fa493f6d612e6c50.
> 
> The maven artifacts are available via repository.apache.org at:
> *https://repository.apache.org/content/repositories/orgapachehadoop-1068/
> *
> 
> We are carrying over the votes from the previous RC given that the delta is
> the license fix.
> 
> Given the above - we are also going to stick with the original deadline for
> the vote : ending on Friday 17th November 2017 2pm PT time.
> 
> Thanks,
> -Arun/Subru


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC2)

2017-11-14 Thread Mukul Kumar Singh

+1 (non-binding)

I built from source on Mac OS X 10.13.1 Java 1.8.0_111

- Deployed on a single node cluster.
- Deployed a ViewFS cluster with two hdfs mount points.
- Performed basic sanity checks.
- Performed basic DFS operations.

Thanks,
Mukul

> On 13-Nov-2017, at 3:01 AM, Subru Krishnan  wrote:
> 
> Hi Folks,
> 
> Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will be the
> starting release for Apache Hadoop 2.9.x line - it includes 30 New Features
> with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues since
> 2.8.2.
> 
> More information about the 2.9.0 release plan can be found here:
> *https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
> *
> 
> New RC is available at: http://home.apache.org/~asuresh/hadoop-2.9.0-RC2/
> 
> 
> The RC tag in git is: release-2.9.0-RC2, and the latest commit id is:
> 1eb05c1dd48fbc9e4b375a76f2046a59103bbeb1.
> 
> The maven artifacts are available via repository.apache.org at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1067/
> 
> 
> Please try the release and vote; the vote will run for the usual 5 days,
> ending on Friday 17th November 2017 2pm PT time.
> 
> We want to give a big shout out to Sunil, Varun, Rohith, Wangda, Vrushali
> and Inigo for the extensive testing/validation which helped prepare for
> RC2. Do report your results in this vote as it'll be very useful to the
> entire community.
> 
> Thanks,
> -Subru/Arun


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC1)

2017-10-24 Thread Mukul Kumar Singh

Thanks Junping,

+1 (non-binding)

I built from source on Mac OS X 10.12.6 Java 1.8.0_111

- Deployed on a single node cluster.
- Deployed a ViewFS cluster with two hdfs mount points.
- Performed basic sanity checks.
- Performed basic DFS operations.

Thanks,
Mukul






On 20/10/17, 6:12 AM, "Junping Du"  wrote:

>Hi folks,
> I've created our new release candidate (RC1) for Apache Hadoop 2.8.2.
>
> Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line and 
> will be the latest stable/production release for Apache Hadoop - it includes 
> 315 new fixed issues since 2.8.1 and 69 fixes are marked as blocker/critical 
> issues.
>
>  More information about the 2.8.2 release plan can be found here: 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>
>  New RC is available at: 
> http://home.apache.org/~junping_du/hadoop-2.8.2-RC1
>
>  The RC tag in git is: release-2.8.2-RC1, and the latest commit id is: 
> 66c47f2a01ad9637879e95f80c41f798373828fb
>
>  The maven artifacts are available via 
> repository.apache.org at: 
> https://repository.apache.org/content/repositories/orgapachehadoop-1064
>
>  Please try the release and vote; the vote will run for the usual 5 days, 
> ending on 10/24/2017 6pm PST time.
>
>Thanks,
>
>Junping
>

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org