Re: [VOTE] Release Apache Hadoop 3.0.2 (RC1)

[Lei Xu]

Hi, everybody,

The 3.0.2 vote has passed with 4 binding +1s, 1 non-binding +1 and no -1s.

Thanks everyone for contributing, testing and voting for this release!



On Fri, Apr 20, 2018 at 4:00 PM, Xiao Chen <x...@cloudera.com> wrote:
> Thanks Eddy for the effort!
>
> +1 (binding)
>
> Downloaded src tarball and verified checksums
> Built from src
> Started a pseudo distributed hdfs cluster
> Verified basic hdfs operations work
> Sanity checked webui and logs
>
> Best,
> -Xiao
>
>
> -Xiao
>
> On Fri, Apr 20, 2018 at 1:44 AM, 俊平堵 <junping...@apache.org> wrote:
>>
>> Thanks Lei for the work!
>>
>> +1 (binding), base on following verification work:
>> - built succeed from source
>> - verified signature
>> - deployed a pseudo cluster and run some simple MR jobs (PI, sleep,
>> terasort, etc.)
>> - checked HDFS/YARN daemons' UI
>> - Tried some rolling upgrade related features: MR over DistributedCache,
>> NM
>> Restart with work preserving, etc.
>>
>> Thanks,
>>
>> Junping
>>
>> 2018-04-17 7:59 GMT+08:00 Lei Xu <l...@apache.org>:
>>
>> > Hi, All
>> >
>> > I've created release candidate RC-1 for Apache Hadoop 3.0.2, to
>> > address missing source jars in the maven repository in RC-0.
>> >
>> > Thanks Ajay Kumar for spotting the error.
>> >
>> > Please note: this is an amendment for Apache Hadoop 3.0.1 release to
>> > fix shaded jars in apache maven repository. The codebase of 3.0.2
>> > release is the same as 3.0.1.  New bug fixes will be included in
>> > Apache Hadoop 3.0.3 instead.
>> >
>> > The release page is:
>> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release
>> >
>> > New RC is available at: http://home.apache.org/~lei/hadoop-3.0.2-RC1/
>> >
>> > The git tag is release-3.0.2-RC1, and the latest commit is
>> > 5c141f7c0f24c12cb8704a6ccc1ff8ec991f41ee, which is the same as RC-0.
>> >
>> > The maven artifacts are available at:
>> > https://repository.apache.org/content/repositories/orgapachehadoop-1102/
>> >
>> > Please try the release, especially, *verify the maven artifacts*, and
>> > vote.
>> >
>> > The vote will run 5 days, ending 4/21/2018.
>> >
>> > Here is my +1.
>> >
>> > Best,
>> >
>> > -
>> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
>> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>> >
>> >
>
>



-- 
Lei (Eddy) Xu
Software Engineer, Cloudera

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[VOTE] Release Apache Hadoop 3.0.2 (RC1)

[Lei Xu]

Hi, All

I've created release candidate RC-1 for Apache Hadoop 3.0.2, to
address missing source jars in the maven repository in RC-0.

Thanks Ajay Kumar for spotting the error.

Please note: this is an amendment for Apache Hadoop 3.0.1 release to
fix shaded jars in apache maven repository. The codebase of 3.0.2
release is the same as 3.0.1.  New bug fixes will be included in
Apache Hadoop 3.0.3 instead.

The release page is:
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release

New RC is available at: http://home.apache.org/~lei/hadoop-3.0.2-RC1/

The git tag is release-3.0.2-RC1, and the latest commit is
5c141f7c0f24c12cb8704a6ccc1ff8ec991f41ee, which is the same as RC-0.

The maven artifacts are available at:
https://repository.apache.org/content/repositories/orgapachehadoop-1102/

Please try the release, especially, *verify the maven artifacts*, and vote.

The vote will run 5 days, ending 4/21/2018.

Here is my +1.

Best,

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.2 (RC0)

[Lei Xu]

Ajay, thanks for spotting this.

I am working on fix the deploy.

On Tue, Apr 10, 2018 at 8:32 AM, Ajay Kumar <ajay.ku...@hortonworks.com> wrote:
> Thanks Lie for working on this.
>
>- Downloaded src tarball and verified checksums
>- Built from src on mac with java 1.8.0_111
>- Built a pseudo distributed hdfs cluster
>- Run test mr jobs (pi, dfsio,wordcount
>- Verified basic hdfs operations
>- Basic validation for webui
>
> ** I checked maven artifacts and it seems source jars are not there (checked 
> hadoop-hdfs , hadoop-client). Not sure if they are required for release.
>
>
> On 4/9/18, 4:19 PM, "Xiao Chen" <x...@cloudera.com> wrote:
>
> Thanks Eddy for the effort!
>
> +1 (binding)
>
>- Downloaded src tarball and verified checksums
>- Built from src
>- Started a pseudo distributed hdfs cluster
>- Verified basic hdfs operations work
>- Sanity checked logs / webui
>
> Best,
> -Xiao
>
>
> On Mon, Apr 9, 2018 at 11:28 AM, Eric Payne 
> <erichadoo...@yahoo.com.invalid>
> wrote:
>
> > Thanks a lot for working to produce this release.
> >
> > +1 (binding)
> > Tested the following:
> > - built from source and installed on 6-node pseudo-cluster
> > - tested Capacity Scheduler FairOrderingPolicy and FifoOrderingPolicy to
> > determine that capacity was assigned as expected in each case
> > - tested user weights with FifoOrderingPolicy to ensure that weights 
> were
> > assigned to users as expected.
> >
> > Eric Payne
> >
> >
> >
> >
> >
> >
> > On Friday, April 6, 2018, 1:17:10 PM CDT, Lei Xu <l...@apache.org> 
> wrote:
> >
> >
> >
> >
> >
> > Hi, All
> >
> > I've created release candidate RC-0 for Apache Hadoop 3.0.2.
> >
> > Please note: this is an amendment for Apache Hadoop 3.0.1 release to
> > fix shaded jars in apache maven repository. The codebase of 3.0.2
> > release is the same as 3.0.1.  New bug fixes will be included in
> > Apache Hadoop 3.0.3 instead.
> >
> > The release page is:
> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release
> >
> > New RC is available at: http://home.apache.org/~lei/hadoop-3.0.2-RC0/
> >
> > The git tag is release-3.0.2-RC0, and the latest commit is
> > 5c141f7c0f24c12cb8704a6ccc1ff8ec991f41ee
> >
> > The maven artifacts are available at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1096/
> >
> > Please try the release, especially, *verify the maven artifacts*, and 
> vote.
> >
> > The vote will run 5 days, ending 4/11/2018.
> >
> > Thanks for everyone who helped to spot the error and proposed fixes!
> >
> > -
> > To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
> >
> >
> > -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
> >
>
>



-- 
Lei (Eddy) Xu
Software Engineer, Cloudera

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[VOTE] Release Apache Hadoop 3.0.2 (RC0)

[Lei Xu]

Hi, All

I've created release candidate RC-0 for Apache Hadoop 3.0.2.

Please note: this is an amendment for Apache Hadoop 3.0.1 release to
fix shaded jars in apache maven repository. The codebase of 3.0.2
release is the same as 3.0.1.  New bug fixes will be included in
Apache Hadoop 3.0.3 instead.

The release page is:
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release

New RC is available at: http://home.apache.org/~lei/hadoop-3.0.2-RC0/

The git tag is release-3.0.2-RC0, and the latest commit is
5c141f7c0f24c12cb8704a6ccc1ff8ec991f41ee

The maven artifacts are available at
https://repository.apache.org/content/repositories/orgapachehadoop-1096/

Please try the release, especially, *verify the maven artifacts*, and vote.

The vote will run 5 days, ending 4/11/2018.

Thanks for everyone who helped to spot the error and proposed fixes!

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.1 (RC1)

[Lei Xu]

Hi, All

Thanks Arpit to find that 3.0.1 artifacts missed shaded jars.  I will
create a new 3.0.2 release with the same codebase as 3.0.1, but deploy
it as shaded jars. A new vote thread will start later this week.

After that, 3.0.3 release will be cut from branch-3.0 and includes the
new bug fixes. Please change the target version of new fixes to 3.0.3.
I will also go over JIRAs to fix targeted versions for committed changes.

Best,

On Mon, Apr 2, 2018 at 6:08 PM, Lei Xu <l...@cloudera.com> wrote:
> Hi, Arpit
>
> I followed this instruction
> https://wiki.apache.org/hadoop/HowToRelease
>
> It instructs to use "mvn deploy -Psign -DskipTests -DskipShade".
>
> It seems wrong, given that 3.0.0 is a shaded jar.
> I will do another deploy to fix it.
>
>
> On Mon, Apr 2, 2018 at 5:31 PM, Arpit Agarwal <aagar...@hortonworks.com> 
> wrote:
>> Hi Lei,
>>
>> It looks like the release artefacts have dummy shaded jars. E.g.
>>
>> Repository Path:  
>> /org/apache/hadoop/hadoop-client-runtime/3.0.1/hadoop-client-runtime-3.0.1.jar
>> Uploaded by:  lei
>> Size: 44.47 KB
>> Uploaded Date:Fri Mar 16 2018 15:50:42 GMT-0700 (PDT)
>> Last Modified:Fri Mar 16 2018 15:50:42 GMT-0700 (PDT)
>>
>> https://repository.apache.org/index.html#view-repositories;releases~browsestorage~/org/apache/hadoop/hadoop-client-runtime/3.0.1/hadoop-client-runtime-3.0.1.jar
>>
>> Am I looking at this wrong or is this supposed to be the shaded jar which is 
>> ~20MB?
>>
>> Thanks,
>> Arpit
>>
>>
>>
>> On 3/23/18, 10:18 AM, "Lei Xu" <l...@cloudera.com> wrote:
>>
>> Hi, All
>>
>> Thanks everyone for voting! The vote passes successfully with 6
>> binding +1s, 7 non-binding +1s and no -1s.
>>
>> I will work on the staging and releases.
>>
>> Best,
>>
>>
>> On Fri, Mar 23, 2018 at 5:10 AM, Kuhu Shukla <kshu...@oath.com.invalid> 
>> wrote:
>> > +1 (non-binding)
>> >
>> > Built from source.
>> > Installed on a pseudo distributed cluster.
>> > Ran word count job and basic hdfs commands.
>> >
>> > Thank you for the effort on this release.
>> >
>> > Regards,
>> > Kuhu
>> >
>> > On Thu, Mar 22, 2018 at 5:25 PM, Elek, Marton <e...@apache.org> wrote:
>> >
>> >>
>> >> +1 (non binding)
>> >>
>> >> I did a full build from source code, created a docker container and 
>> did
>> >> various basic level tests with robotframework based automation and
>> >> docker-compose based pseudo clusters[1].
>> >>
>> >> Including:
>> >>
>> >> * Hdfs federation smoke test
>> >> * Basic ViewFS configuration
>> >> * Yarn example jobs
>> >> * Spark example jobs (with and without yarn)
>> >> * Simple hive table creation
>> >>
>> >> Marton
>> >>
>> >>
>> >> [1]: https://github.com/flokkr/runtime-compose
>> >>
>> >> On 03/18/2018 05:11 AM, Lei Xu wrote:
>> >>
>> >>> Hi, all
>> >>>
>> >>> I've created release candidate RC-1 for Apache Hadoop 3.0.1
>> >>>
>> >>> Apache Hadoop 3.0.1 will be the first bug fix release for Apache
>> >>> Hadoop 3.0 release. It includes 49 bug fixes and security fixes, 
>> which
>> >>> include 12
>> >>> blockers and 17 are critical.
>> >>>
>> >>> Please note:
>> >>> * HDFS-12990. Change default NameNode RPC port back to 8020. It makes
>> >>> incompatible changes to Hadoop 3.0.0.  After 3.0.1 releases, Apache
>> >>> Hadoop 3.0.0 will be deprecated due to this change.
>> >>>
>> >>> The release page is:
>> >>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release
>> >>>
>> >>> New RC is available at: http://home.apache.org/~lei/hadoop-3.0.1-RC1/
>> >>>
>> >>> The git tag is release-3.0.1-RC1, and the latest commit is
>> >>> 496dc57cc2e4f4da117f7a8e3840aaeac0c1d2d0
>> >>>
>> >>> The maven artifacts are available at:
>> >>> 
>&g

Re: [VOTE] Release Apache Hadoop 3.0.1 (RC1)

[Lei Xu]

Hi, Arpit

I followed this instruction
https://wiki.apache.org/hadoop/HowToRelease

It instructs to use "mvn deploy -Psign -DskipTests -DskipShade".

It seems wrong, given that 3.0.0 is a shaded jar.
I will do another deploy to fix it.


On Mon, Apr 2, 2018 at 5:31 PM, Arpit Agarwal <aagar...@hortonworks.com> wrote:
> Hi Lei,
>
> It looks like the release artefacts have dummy shaded jars. E.g.
>
> Repository Path:  
> /org/apache/hadoop/hadoop-client-runtime/3.0.1/hadoop-client-runtime-3.0.1.jar
> Uploaded by:  lei
> Size: 44.47 KB
> Uploaded Date:Fri Mar 16 2018 15:50:42 GMT-0700 (PDT)
> Last Modified:Fri Mar 16 2018 15:50:42 GMT-0700 (PDT)
>
> https://repository.apache.org/index.html#view-repositories;releases~browsestorage~/org/apache/hadoop/hadoop-client-runtime/3.0.1/hadoop-client-runtime-3.0.1.jar
>
> Am I looking at this wrong or is this supposed to be the shaded jar which is 
> ~20MB?
>
> Thanks,
> Arpit
>
>
>
> On 3/23/18, 10:18 AM, "Lei Xu" <l...@cloudera.com> wrote:
>
> Hi, All
>
> Thanks everyone for voting! The vote passes successfully with 6
> binding +1s, 7 non-binding +1s and no -1s.
>
> I will work on the staging and releases.
>
> Best,
>
>
> On Fri, Mar 23, 2018 at 5:10 AM, Kuhu Shukla <kshu...@oath.com.invalid> 
> wrote:
> > +1 (non-binding)
> >
> > Built from source.
> > Installed on a pseudo distributed cluster.
> > Ran word count job and basic hdfs commands.
> >
> > Thank you for the effort on this release.
> >
> > Regards,
> > Kuhu
> >
> > On Thu, Mar 22, 2018 at 5:25 PM, Elek, Marton <e...@apache.org> wrote:
> >
> >>
> >> +1 (non binding)
> >>
> >> I did a full build from source code, created a docker container and did
> >> various basic level tests with robotframework based automation and
> >> docker-compose based pseudo clusters[1].
> >>
> >> Including:
> >>
> >> * Hdfs federation smoke test
> >> * Basic ViewFS configuration
> >> * Yarn example jobs
> >> * Spark example jobs (with and without yarn)
> >> * Simple hive table creation
> >>
> >> Marton
> >>
> >>
> >> [1]: https://github.com/flokkr/runtime-compose
> >>
> >> On 03/18/2018 05:11 AM, Lei Xu wrote:
> >>
> >>> Hi, all
> >>>
> >>> I've created release candidate RC-1 for Apache Hadoop 3.0.1
> >>>
> >>> Apache Hadoop 3.0.1 will be the first bug fix release for Apache
> >>> Hadoop 3.0 release. It includes 49 bug fixes and security fixes, which
> >>> include 12
> >>> blockers and 17 are critical.
> >>>
> >>> Please note:
> >>> * HDFS-12990. Change default NameNode RPC port back to 8020. It makes
> >>> incompatible changes to Hadoop 3.0.0.  After 3.0.1 releases, Apache
> >>> Hadoop 3.0.0 will be deprecated due to this change.
> >>>
> >>> The release page is:
> >>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release
> >>>
> >>> New RC is available at: http://home.apache.org/~lei/hadoop-3.0.1-RC1/
> >>>
> >>> The git tag is release-3.0.1-RC1, and the latest commit is
> >>> 496dc57cc2e4f4da117f7a8e3840aaeac0c1d2d0
> >>>
> >>> The maven artifacts are available at:
> >>> 
> https://repository.apache.org/content/repositories/orgapachehadoop-1081/
> >>>
> >>> Please try the release and vote; the vote will run for the usual 5
> >>> days, ending on 3/22/2017 6pm PST time.
> >>>
> >>> Thanks!
> >>>
> >>> -
> >>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> >>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> >>>
> >>>
> >> -
> >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >>
> >>
>
>
>
> --
> Lei (Eddy) Xu
> Software Engineer, Cloudera
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>
>



-- 
Lei (Eddy) Xu
Software Engineer, Cloudera

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Adopt HDSL as a new Hadoop subproject

[Lei Xu]

+1

Best,

On Mon, Mar 26, 2018 at 10:38 AM, Xiao Chen  wrote:
> +1
>
> Thanks,
> -Xiao
>
> On Sun, Mar 25, 2018 at 9:07 PM, Akira Ajisaka 
> wrote:
>
>> +1
>>
>> Thanks,
>> Akira
>>
>>
>> On 2018/03/24 15:18, Lokesh Jain wrote:
>>
>>> +1 (non-binding)
>>>
>>> Thanks
>>> Lokesh
>>>
>>>
>>> -
>>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>>>
>>>
>> -
>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>>
>>



-- 
Lei (Eddy) Xu
Software Engineer, Cloudera

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.1 (RC1)

[Lei Xu]

Hi, All

Thanks everyone for voting! The vote passes successfully with 6
binding +1s, 7 non-binding +1s and no -1s.

I will work on the staging and releases.

Best,


On Fri, Mar 23, 2018 at 5:10 AM, Kuhu Shukla <kshu...@oath.com.invalid> wrote:
> +1 (non-binding)
>
> Built from source.
> Installed on a pseudo distributed cluster.
> Ran word count job and basic hdfs commands.
>
> Thank you for the effort on this release.
>
> Regards,
> Kuhu
>
> On Thu, Mar 22, 2018 at 5:25 PM, Elek, Marton <e...@apache.org> wrote:
>
>>
>> +1 (non binding)
>>
>> I did a full build from source code, created a docker container and did
>> various basic level tests with robotframework based automation and
>> docker-compose based pseudo clusters[1].
>>
>> Including:
>>
>> * Hdfs federation smoke test
>> * Basic ViewFS configuration
>> * Yarn example jobs
>> * Spark example jobs (with and without yarn)
>> * Simple hive table creation
>>
>> Marton
>>
>>
>> [1]: https://github.com/flokkr/runtime-compose
>>
>> On 03/18/2018 05:11 AM, Lei Xu wrote:
>>
>>> Hi, all
>>>
>>> I've created release candidate RC-1 for Apache Hadoop 3.0.1
>>>
>>> Apache Hadoop 3.0.1 will be the first bug fix release for Apache
>>> Hadoop 3.0 release. It includes 49 bug fixes and security fixes, which
>>> include 12
>>> blockers and 17 are critical.
>>>
>>> Please note:
>>> * HDFS-12990. Change default NameNode RPC port back to 8020. It makes
>>> incompatible changes to Hadoop 3.0.0.  After 3.0.1 releases, Apache
>>> Hadoop 3.0.0 will be deprecated due to this change.
>>>
>>> The release page is:
>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release
>>>
>>> New RC is available at: http://home.apache.org/~lei/hadoop-3.0.1-RC1/
>>>
>>> The git tag is release-3.0.1-RC1, and the latest commit is
>>> 496dc57cc2e4f4da117f7a8e3840aaeac0c1d2d0
>>>
>>> The maven artifacts are available at:
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1081/
>>>
>>> Please try the release and vote; the vote will run for the usual 5
>>> days, ending on 3/22/2017 6pm PST time.
>>>
>>> Thanks!
>>>
>>> -
>>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>>>
>>>
>> -
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>
>>



-- 
Lei (Eddy) Xu
Software Engineer, Cloudera

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Adopt HDSL as a new Hadoop subproject

[Lei Xu]

Hi, Owen

Thanks a lot for this proposal, as I believe it has addressed most of
the concerns of the community.

I have one concrete question about how this HDSL subproject being
separated: Ozone / HDSL was designed in the current way to re-use the
existing HDFS code base as much as possible, thus today for this
container service is in DataNode itself.  It is not clear to me after
separate HDSL into a new project, where would be these code? And if it
is still in DataNode somehow (logically?), how to sync changes between
these two projects even the code is not physically co-located.

I would +1 if this concern will be addressed.

Best,


On Thu, Mar 22, 2018 at 10:35 AM, Chris Douglas  wrote:
> On Thu, Mar 22, 2018 at 10:23 AM, Andrew Wang  
> wrote:
>> We want the git hash to match the contents of the tarball and tag, which is
>> beyond what create release does right now. It doesn't do any git stuff.
>
> ...and it can't? Even if this remains a manual step, it's not a
> significant concession.
>
>> If this vote to adopt a new project also includes merging to trunk (it
>> sounds like it?), I feel like we should settle these questions first.
>
> No, this is just measuring consensus so effort arranging the merge is
> well-spent. The merge vote will come later. -C
>
>> Best,
>> Andrew
>>
>> On Mar 22, 2018 9:51 AM, "Chris Douglas"  wrote:
>>
>> +1 (binding)
>>
>> This compromise seems to address most of the concerns raised during
>> the discussion. Thanks for proposing and driving this, Owen.
>>
>> On Thu, Mar 22, 2018 at 9:30 AM, Andrew Wang 
>> wrote:
>>> In Owen's proposal, it says to delete the module from the release branch.
>>> We need to do this since the source tarball is our official Apache release
>>> artifact, the rest are convenience binaries. So the Maven profile is
>>> insufficient for this.
>>
>> Eliminating manual steps to create a release is desirable, but
>> privileging it above all the development efficiencies gained by
>> merging to the same repo... we don't cut releases that often.
>>
>> Moreover, the steps to remove the module don't need to be manual. Once
>> we work out the steps, would you be willing to update the
>> create-release script? -C
>>
>>
>> On Tue, Mar 20, 2018 at 11:20 AM, Owen O'Malley 
>> wrote:
>>> All,
>>>
>>> Following our discussions on the previous thread (Merging branch HDFS-7240
>>> to trunk), I'd like to propose the following:
>>>
>>> * HDSL become a subproject of Hadoop.
>>> * HDSL will release separately from Hadoop. Hadoop releases will not
>>> contain HDSL and vice versa.
>>> * HDSL will get its own jira instance so that the release tags stay
>>> separate.
>>> * On trunk (as opposed to release branches) HDSL will be a separate module
>>> in Hadoop's source tree. This will enable the HDSL to work on their trunk
>>> and the Hadoop trunk without making releases for every change.
>>> * Hadoop's trunk will only build HDSL if a non-default profile is enabled.
>>> * When Hadoop creates a release branch, the RM will delete the HDSL module
>>> from the branch.
>>> * HDSL will have their own Yetus checks and won't cause failures in the
>>> Hadoop patch check.
>>>
>>> I think this accomplishes most of the goals of encouraging HDSL
>>> development
>>> while minimizing the potential for disruption of HDFS development.
>>>
>>> The vote will run the standard 7 days and requires a lazy 2/3 vote. PMC
>>> votes are binding, but everyone is encouraged to vote.
>>>
>>> +1 (binding)
>>>
>>> .. Owen
>>
>> -
>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>>
>>
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>



-- 
Lei (Eddy) Xu
Software Engineer, Cloudera

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.1 (RC1)

[Lei Xu]

Sure, Akira

.mds files are uploaded to http://home.apache.org/~lei/hadoop-3.0.1-RC1/

On Sun, Mar 18, 2018 at 6:04 PM, Akira Ajisaka
<ajisa...@oss.nttdata.co.jp> wrote:
> Hi Lei,
>
> Would you provide SHA checksum files instead of MD5?
> http://www.apache.org/dev/release-distribution#sigs-and-sums
>
> -Akira
>
>
> On 2018/03/18 13:11, Lei Xu wrote:
>>
>> Hi, all
>>
>> I've created release candidate RC-1 for Apache Hadoop 3.0.1
>>
>> Apache Hadoop 3.0.1 will be the first bug fix release for Apache
>> Hadoop 3.0 release. It includes 49 bug fixes and security fixes, which
>> include 12
>> blockers and 17 are critical.
>>
>> Please note:
>> * HDFS-12990. Change default NameNode RPC port back to 8020. It makes
>> incompatible changes to Hadoop 3.0.0.  After 3.0.1 releases, Apache
>> Hadoop 3.0.0 will be deprecated due to this change.
>>
>> The release page is:
>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release
>>
>> New RC is available at: http://home.apache.org/~lei/hadoop-3.0.1-RC1/
>>
>> The git tag is release-3.0.1-RC1, and the latest commit is
>> 496dc57cc2e4f4da117f7a8e3840aaeac0c1d2d0
>>
>> The maven artifacts are available at:
>> https://repository.apache.org/content/repositories/orgapachehadoop-1081/
>>
>> Please try the release and vote; the vote will run for the usual 5
>> days, ending on 3/22/2017 6pm PST time.
>>
>> Thanks!
>>
>> -
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>
>



-- 
Lei (Eddy) Xu
Software Engineer, Cloudera

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[VOTE] Release Apache Hadoop 3.0.1 (RC1)

[Lei Xu]

Hi, all

I've created release candidate RC-1 for Apache Hadoop 3.0.1

Apache Hadoop 3.0.1 will be the first bug fix release for Apache
Hadoop 3.0 release. It includes 49 bug fixes and security fixes, which
include 12
blockers and 17 are critical.

Please note:
* HDFS-12990. Change default NameNode RPC port back to 8020. It makes
incompatible changes to Hadoop 3.0.0.  After 3.0.1 releases, Apache
Hadoop 3.0.0 will be deprecated due to this change.

The release page is:
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release

New RC is available at: http://home.apache.org/~lei/hadoop-3.0.1-RC1/

The git tag is release-3.0.1-RC1, and the latest commit is
496dc57cc2e4f4da117f7a8e3840aaeac0c1d2d0

The maven artifacts are available at:
https://repository.apache.org/content/repositories/orgapachehadoop-1081/

Please try the release and vote; the vote will run for the usual 5
days, ending on 3/22/2017 6pm PST time.

Thanks!

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.1 (RC0)

[Lei Xu]

Hi, All

Thank you so much for the votes so far!
I will make a new RC1 in the next few days to include the latest
fixes. Please wait and vote for the new RC1.

Best regards,

On Thu, Feb 22, 2018 at 1:17 PM, Ajay Kumar <ajay.ku...@hortonworks.com> wrote:
> +1 non-binding
>
>
>
> Build from source in CentOs7.
> Deployed 4 node secure federated cluster (HDFS HA,Non-HA YARN)
> Run TestDFSIO(write/read), pi and wordcount.
> Verified basic hdfs command
>
>
>
> Thanks,
>
> Ajay
>
>
>
> On 2/20/18, 11:16 PM, "Vinayakumar B" <vinayakum...@apache.org> wrote:
>
>
>
> +1 Binding,
>
>
>
> 1. Checked out code from tag.
>
> 2. Built and deployed a 3-node docker cluster in secure mode (HDFS HA,
>
> Non-HA YARN). Docker script available in [1], if anyone need,
>
> 3. Ran wordcount job successfully.
>
> 4. Verified basic webhdfs commands using curl.
>
> 5. Verified basic HDFS commands
>
>
>
>
>
>     [1] https://github.com/vinayakumarb/docker-cluster-deployer
>
>
>
> Thanks,
>
> -Vinay
>
>
>
> On Wed, Feb 21, 2018 at 5:41 AM, Lei Xu <l...@cloudera.com> wrote:
>
>
>
> > Dear PMC and committers
>
> >
>
>     > Please help to verify and vote this release. I will extend the voting
>
> > period to Friday (Feb 23th) of this week.
>
> >
>
> > Thanks!
>
> >
>
> > On Thu, Feb 15, 2018 at 3:36 PM, Lei Xu <l...@apache.org> wrote:
>
> > > Hi, all
>
> > >
>
> > > I've created release candidate 0 for Apache Hadoop 3.0.1
>
> > >
>
> > > Apache Hadoop 3.0.1 will be the first bug fix release for Apache
>
> > > Hadoop 3.0 release. It includes 49 bug fixes, which include 10
>
> > > blockers and 8 are critical.
>
> > >
>
> > > Please note:
>
> > > * HDFS-12990. Change default NameNode RPC port back to 8020. It
> makes
>
> > > incompatible changes to Hadoop 3.0.0.  After 3.0.1 releases, Apache
>
> > > Hadoop 3.0.0 will be deprecated due to this change.
>
> > >
>
> > > The release page is:
>
> > >
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release
>
> > >
>
> > > New RC is available at:
> http://home.apache.org/~lei/hadoop-3.0.1-RC0/
>
> > >
>
> > > The git tag is release-3.0.1-RC0, and the latest commit is
>
> > > 494d075055b52b0cc922bc25237e231bb3771c90
>
> > >
>
> > > The maven artifacts are available:
>
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1078/
>
> > >
>
> > > Please try the release and vote; the vote will run for the usual 5
>
> > > days, ending on 2/20/2017 6pm PST time.
>
> > >
>
> > > Thanks!
>
> > >
>
> > > --
>
>> > Lei (Eddy) Xu
>
> >
>
> >
>
> >
>
> > --
>
> > Lei (Eddy) Xu
>
> > Software Engineer, Cloudera
>
> >
>
> > -
>
> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
>
> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
> >
>
> >
>
>



-- 
Lei (Eddy) Xu
Software Engineer, Cloudera

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.1 (RC0)

[Lei Xu]

Dear PMC and committers

Please help to verify and vote this release. I will extend the voting
period to Friday (Feb 23th) of this week.

Thanks!

On Thu, Feb 15, 2018 at 3:36 PM, Lei Xu <l...@apache.org> wrote:
> Hi, all
>
> I've created release candidate 0 for Apache Hadoop 3.0.1
>
> Apache Hadoop 3.0.1 will be the first bug fix release for Apache
> Hadoop 3.0 release. It includes 49 bug fixes, which include 10
> blockers and 8 are critical.
>
> Please note:
> * HDFS-12990. Change default NameNode RPC port back to 8020. It makes
> incompatible changes to Hadoop 3.0.0.  After 3.0.1 releases, Apache
> Hadoop 3.0.0 will be deprecated due to this change.
>
> The release page is:
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release
>
> New RC is available at: http://home.apache.org/~lei/hadoop-3.0.1-RC0/
>
> The git tag is release-3.0.1-RC0, and the latest commit is
> 494d075055b52b0cc922bc25237e231bb3771c90
>
> The maven artifacts are available:
> https://repository.apache.org/content/repositories/orgapachehadoop-1078/
>
> Please try the release and vote; the vote will run for the usual 5
> days, ending on 2/20/2017 6pm PST time.
>
> Thanks!
>
> --
> Lei (Eddy) Xu



-- 
Lei (Eddy) Xu
Software Engineer, Cloudera

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[VOTE] Release Apache Hadoop 3.0.1 (RC0)

[Lei Xu]

Hi, all

I've created release candidate 0 for Apache Hadoop 3.0.1

Apache Hadoop 3.0.1 will be the first bug fix release for Apache
Hadoop 3.0 release. It includes 49 bug fixes, which include 10
blockers and 8 are critical.

Please note:
* HDFS-12990. Change default NameNode RPC port back to 8020. It makes
incompatible changes to Hadoop 3.0.0.  After 3.0.1 releases, Apache
Hadoop 3.0.0 will be deprecated due to this change.

The release page is:
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release

New RC is available at: http://home.apache.org/~lei/hadoop-3.0.1-RC0/

The git tag is release-3.0.1-RC0, and the latest commit is
494d075055b52b0cc922bc25237e231bb3771c90

The maven artifacts are available:
https://repository.apache.org/content/repositories/orgapachehadoop-1078/

Please try the release and vote; the vote will run for the usual 5
days, ending on 2/20/2017 6pm PST time.

Thanks!

-- 
Lei (Eddy) Xu

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Apache Hadoop 3.0.1 Release plan

[Lei Xu]

Hi, Brahma

Thanks for reminder. YARN-5742 does not look like a blocker to me. I
will create a RC right after HADOOP-14060.

On Thu, Feb 8, 2018 at 7:35 AM, Kihwal Lee <kih...@oath.com> wrote:
> HADOOP-14060 is a blocker.  Daryn will add more detail to the jira or to
> this thread.
>
> On Thu, Feb 8, 2018 at 7:01 AM, Brahma Reddy Battula <brbapa...@gmail.com>
> wrote:
>>
>> Hi Eddy,
>>
>> HDFS-12990 got committed to 3.0.1,can we have RC for 3.0.1 (only
>> YARN-5742
>> blocker is open )  ?
>>
>>
>> On Sat, Feb 3, 2018 at 12:40 AM, Chris Douglas <cdoug...@apache.org>
>> wrote:
>>
>> > On Fri, Feb 2, 2018 at 10:22 AM, Arpit Agarwal
>> > <aagar...@hortonworks.com>
>> > wrote:
>> > > Do you plan to roll an RC with an uncommitted fix? That isn't the
>> > > right
>> > approach.
>> >
>> > The fix will be committed to the release branch. We'll vote on the
>> > release, and if it receives a majority of +1 votes then it becomes
>> > 3.0.1. That's how the PMC decides how to move forward. In this case,
>> > that will also resolve whether or not it can be committed to trunk.
>> >
>> > If this logic is unpersuasive, then we can require a 2/3 majority to
>> > replace the codebase. Either way, the PMC will vote to define the
>> > consensus view when it is not emergent.
>> >
>> > > This issue has good visibility and enough discussion.
>> >
>> > Yes, it has. We always prefer consensus to voting, but when discussion
>> > reveals that complete consensus is impossible, we still need a way
>> > forward. This is rare, and usually reserved for significant changes
>> > (like merging YARN). Frankly, it's embarrassing to resort to it here,
>> > but here we are.
>> >
>> > > If there is a binding veto in effect then the change must be
>> > > abandoned.
>> > Else you should be able to proceed with committing. However, 3.0.0 must
>> > be
>> > called out as an abandoned release if we commit it.
>> >
>> > This is not accurate. A binding veto from any committer halts
>> > progress, but the PMC sets the direction of the project. That includes
>> > making decisions that are not universally accepted. -C
>> >
>> > > On 2/1/18, 3:01 PM, "Lei Xu" <l...@cloudera.com> wrote:
>> > >
>> > > Sounds good to me, ATM.
>> > >
>> > > On Thu, Feb 1, 2018 at 2:34 PM, Aaron T. Myers <a...@apache.org>
>> > wrote:
>> > > > Hey Anu,
>> > > >
>> > > > My feeling on HDFS-12990 is that we've discussed it quite a bit
>> > already and
>> > > > it doesn't seem at this point like either side is going to
>> > > budge.
>> > I'm
>> > > > certainly happy to have a phone call about it, but I don't
>> > > expect
>> > that we'd
>> > > > make much progress.
>> > > >
>> > > > My suggestion is that we simply include the patch posted to
>> > HDFS-12990 in
>> > > > the 3.0.1 RC and call this issue out clearly in the subsequent
>> > VOTE thread
>> > > > for the 3.0.1 release. Eddy, are you up for that?
>> > > >
>> > > > Best,
>> > > > Aaron
>> > > >
>> > > > On Thu, Feb 1, 2018 at 1:13 PM, Lei Xu <l...@cloudera.com> wrote:
>> > > >>
>> > > >> +Xiao
>> > >     >>
>> > > >> My understanding is that we will have this for 3.0.1.   Xiao,
>> > could
>> > > >> you give your inputs here?
>> > > >>
>> > > >> On Thu, Feb 1, 2018 at 11:55 AM, Anu Engineer <
>> > aengin...@hortonworks.com>
>> > > >> wrote:
>> > > >> > Hi Eddy,
>> > >     >> >
>> > > >> > Thanks for driving this release. Just a quick question, do we
>> > have time
>> > > >> > to close this issue?
>> > > >> > https://issues.apache.org/jira/browse/HDFS-12990
>> > > >> >
>> > > >> > or are we abandoning it? I believe that this is the last
>> > > window
>> > for us
>> > > >> > to fix this issue.
>> > > >> >
>

Re: Apache Hadoop 3.0.1 Release plan

[Lei Xu]

Sounds good to me, ATM.

On Thu, Feb 1, 2018 at 2:34 PM, Aaron T. Myers <a...@apache.org> wrote:
> Hey Anu,
>
> My feeling on HDFS-12990 is that we've discussed it quite a bit already and
> it doesn't seem at this point like either side is going to budge. I'm
> certainly happy to have a phone call about it, but I don't expect that we'd
> make much progress.
>
> My suggestion is that we simply include the patch posted to HDFS-12990 in
> the 3.0.1 RC and call this issue out clearly in the subsequent VOTE thread
> for the 3.0.1 release. Eddy, are you up for that?
>
> Best,
> Aaron
>
> On Thu, Feb 1, 2018 at 1:13 PM, Lei Xu <l...@cloudera.com> wrote:
>>
>> +Xiao
>>
>> My understanding is that we will have this for 3.0.1.   Xiao, could
>> you give your inputs here?
>>
>> On Thu, Feb 1, 2018 at 11:55 AM, Anu Engineer <aengin...@hortonworks.com>
>> wrote:
>> > Hi Eddy,
>> >
>> > Thanks for driving this release. Just a quick question, do we have time
>> > to close this issue?
>> > https://issues.apache.org/jira/browse/HDFS-12990
>> >
>> > or are we abandoning it? I believe that this is the last window for us
>> > to fix this issue.
>> >
>> > Should we have a call and get this resolved one way or another?
>> >
>> > Thanks
>> > Anu
>> >
>> > On 2/1/18, 10:51 AM, "Lei Xu" <l...@cloudera.com> wrote:
>> >
>> > Hi, All
>> >
>> > I just cut branch-3.0.1 from branch-3.0.  Please make sure all
>> > patches
>> > targeted to 3.0.1 being checked in both branch-3.0 and branch-3.0.1.
>> >
>> > Thanks!
>> > Eddy
>> >
>> > On Tue, Jan 9, 2018 at 11:17 AM, Lei Xu <l...@cloudera.com> wrote:
>> > > Hi, All
>> > >
>> > > We have released Apache Hadoop 3.0.0 in December [1]. To further
>> > > improve the quality of release, we plan to cut branch-3.0.1 branch
>> > > tomorrow for the preparation of Apache Hadoop 3.0.1 release. The
>> > focus
>> > > of 3.0.1 will be fixing blockers (3), critical bugs (1) and bug
>> > fixes
>> > > [2].  No new features and improvement should be included.
>> > >
>> > > We plan to cut branch-3.0.1 tomorrow (Jan 10th) and vote for RC on
>> > Feb
>> > > 1st, targeting for Feb 9th release.
>> > >
>> > > Please feel free to share your insights.
>> > >
>> > > [1]
>> > https://www.mail-archive.com/general@hadoop.apache.org/msg07757.html
>> > > [2] https://issues.apache.org/jira/issues/?filter=12342842
>> > >
>> > > Best,
>> > > --
>> > > Lei (Eddy) Xu
>> > > Software Engineer, Cloudera
>> >
>> >
>> >
>> > --
>> > Lei (Eddy) Xu
>> > Software Engineer, Cloudera
>> >
>> >
>> > -
>> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>> >
>> >
>> >
>>
>>
>>
>> --
>> Lei (Eddy) Xu
>> Software Engineer, Cloudera
>>
>> -
>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>>
>



-- 
Lei (Eddy) Xu
Software Engineer, Cloudera

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Apache Hadoop 3.0.1 Release plan

[Lei Xu]

+Xiao

My understanding is that we will have this for 3.0.1.   Xiao, could
you give your inputs here?

On Thu, Feb 1, 2018 at 11:55 AM, Anu Engineer <aengin...@hortonworks.com> wrote:
> Hi Eddy,
>
> Thanks for driving this release. Just a quick question, do we have time to 
> close this issue?
> https://issues.apache.org/jira/browse/HDFS-12990
>
> or are we abandoning it? I believe that this is the last window for us to fix 
> this issue.
>
> Should we have a call and get this resolved one way or another?
>
> Thanks
> Anu
>
> On 2/1/18, 10:51 AM, "Lei Xu" <l...@cloudera.com> wrote:
>
> Hi, All
>
> I just cut branch-3.0.1 from branch-3.0.  Please make sure all patches
> targeted to 3.0.1 being checked in both branch-3.0 and branch-3.0.1.
>
> Thanks!
> Eddy
>
> On Tue, Jan 9, 2018 at 11:17 AM, Lei Xu <l...@cloudera.com> wrote:
> > Hi, All
> >
> > We have released Apache Hadoop 3.0.0 in December [1]. To further
> > improve the quality of release, we plan to cut branch-3.0.1 branch
> > tomorrow for the preparation of Apache Hadoop 3.0.1 release. The focus
> > of 3.0.1 will be fixing blockers (3), critical bugs (1) and bug fixes
> > [2].  No new features and improvement should be included.
> >
> > We plan to cut branch-3.0.1 tomorrow (Jan 10th) and vote for RC on Feb
> > 1st, targeting for Feb 9th release.
> >
> > Please feel free to share your insights.
> >
> > [1] https://www.mail-archive.com/general@hadoop.apache.org/msg07757.html
> > [2] https://issues.apache.org/jira/issues/?filter=12342842
> >
> > Best,
> > --
> > Lei (Eddy) Xu
> > Software Engineer, Cloudera
>
>
>
> --
> Lei (Eddy) Xu
> Software Engineer, Cloudera
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>
>



-- 
Lei (Eddy) Xu
Software Engineer, Cloudera

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Apache Hadoop 3.0.1 Release plan

[Lei Xu]

Hi, All

I just cut branch-3.0.1 from branch-3.0.  Please make sure all patches
targeted to 3.0.1 being checked in both branch-3.0 and branch-3.0.1.

Thanks!
Eddy

On Tue, Jan 9, 2018 at 11:17 AM, Lei Xu <l...@cloudera.com> wrote:
> Hi, All
>
> We have released Apache Hadoop 3.0.0 in December [1]. To further
> improve the quality of release, we plan to cut branch-3.0.1 branch
> tomorrow for the preparation of Apache Hadoop 3.0.1 release. The focus
> of 3.0.1 will be fixing blockers (3), critical bugs (1) and bug fixes
> [2].  No new features and improvement should be included.
>
> We plan to cut branch-3.0.1 tomorrow (Jan 10th) and vote for RC on Feb
> 1st, targeting for Feb 9th release.
>
> Please feel free to share your insights.
>
> [1] https://www.mail-archive.com/general@hadoop.apache.org/msg07757.html
> [2] https://issues.apache.org/jira/issues/?filter=12342842
>
> Best,
> --
> Lei (Eddy) Xu
> Software Engineer, Cloudera



-- 
Lei (Eddy) Xu
Software Engineer, Cloudera

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Apache Hadoop 3.0.1 Release plan

[Lei Xu]

Hi, Andrew and Jason

Thanks for the feedback!

I will follow the pattern and cut the branch on Feb 1st and start the
RC vote around Feb 5th then.

Best

On Tue, Jan 9, 2018 at 11:35 AM, Jason Lowe <jl...@oath.com> wrote:
> Is it necessary to cut the branch so far ahead of the release?  branch-3.0
> is already a maintenance line for 3.0.x releases.  Is there a known
> feature/improvement planned to go into branch-3.0 that is not desirable for
> the 3.0.1 release?
>
> I have found in the past that branching so early leads to many useful fixes
> being unnecessarily postponed to future releases because committers forget
> to pick to the new, relatively long-lived patch branch.  This becomes
> especially true if blockers end up dragging out the ultimate release date,
> which has historically been quite common.  My preference would be to cut
> this branch as close to the RC as possible.
>
> Jason
>
>
> On Tue, Jan 9, 2018 at 1:17 PM, Lei Xu <l...@cloudera.com> wrote:
>>
>> Hi, All
>>
>> We have released Apache Hadoop 3.0.0 in December [1]. To further
>> improve the quality of release, we plan to cut branch-3.0.1 branch
>> tomorrow for the preparation of Apache Hadoop 3.0.1 release. The focus
>> of 3.0.1 will be fixing blockers (3), critical bugs (1) and bug fixes
>> [2].  No new features and improvement should be included.
>>
>> We plan to cut branch-3.0.1 tomorrow (Jan 10th) and vote for RC on Feb
>> 1st, targeting for Feb 9th release.
>>
>> Please feel free to share your insights.
>>
>> [1] https://www.mail-archive.com/general@hadoop.apache.org/msg07757.html
>> [2] https://issues.apache.org/jira/issues/?filter=12342842
>>
>> Best,
>> --
>> Lei (Eddy) Xu
>> Software Engineer, Cloudera
>>
>> -
>> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>>
>



-- 
Lei (Eddy) Xu
Software Engineer, Cloudera

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Apache Hadoop 3.0.1 Release plan

[Lei Xu]

Hi, Andrew and Jason

Thanks for the feedback!

I will follow the pattern and cut the branch on Feb 1st and start the
RC vote around Feb 5th then.

Best

On Tue, Jan 9, 2018 at 11:35 AM, Jason Lowe <jl...@oath.com> wrote:
> Is it necessary to cut the branch so far ahead of the release?  branch-3.0
> is already a maintenance line for 3.0.x releases.  Is there a known
> feature/improvement planned to go into branch-3.0 that is not desirable for
> the 3.0.1 release?
>
> I have found in the past that branching so early leads to many useful fixes
> being unnecessarily postponed to future releases because committers forget
> to pick to the new, relatively long-lived patch branch.  This becomes
> especially true if blockers end up dragging out the ultimate release date,
> which has historically been quite common.  My preference would be to cut
> this branch as close to the RC as possible.
>
> Jason
>
>
> On Tue, Jan 9, 2018 at 1:17 PM, Lei Xu <l...@cloudera.com> wrote:
>>
>> Hi, All
>>
>> We have released Apache Hadoop 3.0.0 in December [1]. To further
>> improve the quality of release, we plan to cut branch-3.0.1 branch
>> tomorrow for the preparation of Apache Hadoop 3.0.1 release. The focus
>> of 3.0.1 will be fixing blockers (3), critical bugs (1) and bug fixes
>> [2].  No new features and improvement should be included.
>>
>> We plan to cut branch-3.0.1 tomorrow (Jan 10th) and vote for RC on Feb
>> 1st, targeting for Feb 9th release.
>>
>> Please feel free to share your insights.
>>
>> [1] https://www.mail-archive.com/general@hadoop.apache.org/msg07757.html
>> [2] https://issues.apache.org/jira/issues/?filter=12342842
>>
>> Best,
>> --
>> Lei (Eddy) Xu
>> Software Engineer, Cloudera
>>
>> -
>> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>>
>



-- 
Lei (Eddy) Xu
Software Engineer, Cloudera

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop 3.0.1 Release plan

[Lei Xu]

Hi, All

We have released Apache Hadoop 3.0.0 in December [1]. To further
improve the quality of release, we plan to cut branch-3.0.1 branch
tomorrow for the preparation of Apache Hadoop 3.0.1 release. The focus
of 3.0.1 will be fixing blockers (3), critical bugs (1) and bug fixes
[2].  No new features and improvement should be included.

We plan to cut branch-3.0.1 tomorrow (Jan 10th) and vote for RC on Feb
1st, targeting for Feb 9th release.

Please feel free to share your insights.

[1] https://www.mail-archive.com/general@hadoop.apache.org/msg07757.html
[2] https://issues.apache.org/jira/issues/?filter=12342842

Best,
-- 
Lei (Eddy) Xu
Software Engineer, Cloudera

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC1

[Lei Xu]

+1 (binding)

* Verified src tarball and bin tarball, verified md5 of each.
* Build source with -Pdist,native
* Started a pseudo cluster
* Run ec -listPolicies / -getPolicy / -setPolicy on /  , and run hdfs
dfs put/get/cat on "/" with XOR-2-1 policy.

Thanks Andrew for this great effort!

Best,


On Tue, Dec 12, 2017 at 9:55 AM, Andrew Wang  wrote:
> Hi Wei-Chiu,
>
> The patchprocess directory is left over from the create-release process,
> and it looks empty to me. We should still file a create-release JIRA to fix
> this, but I think this is not a blocker. Would you agree?
>
> Best,
> Andrew
>
> On Tue, Dec 12, 2017 at 9:44 AM, Wei-Chiu Chuang 
> wrote:
>
>> Hi Andrew, thanks the tremendous effort.
>> I found an empty "patchprocess" directory in the source tarball, that is
>> not there if you clone from github. Any chance you might have some leftover
>> trash when you made the tarball?
>> Not wanting to nitpicking, but you might want to double check so we don't
>> ship anything private to you in public :)
>>
>>
>>
>> On Tue, Dec 12, 2017 at 7:48 AM, Ajay Kumar 
>> wrote:
>>
>>> +1 (non-binding)
>>> Thanks for driving this, Andrew Wang!!
>>>
>>> - downloaded the src tarball and verified md5 checksum
>>> - built from source with jdk 1.8.0_111-b14
>>> - brought up a pseudo distributed cluster
>>> - did basic file system operations (mkdir, list, put, cat) and
>>> confirmed that everything was working
>>> - Run word count, pi and DFSIOTest
>>> - run hdfs and yarn, confirmed that the NN, RM web UI worked
>>>
>>> Cheers,
>>> Ajay
>>>
>>> On 12/11/17, 9:35 PM, "Xiao Chen"  wrote:
>>>
>>> +1 (binding)
>>>
>>> - downloaded src tarball, verified md5
>>> - built from source with jdk1.8.0_112
>>> - started a pseudo cluster with hdfs and kms
>>> - sanity checked encryption related operations working
>>> - sanity checked webui and logs.
>>>
>>> -Xiao
>>>
>>> On Mon, Dec 11, 2017 at 6:10 PM, Aaron T. Myers 
>>> wrote:
>>>
>>> > +1 (binding)
>>> >
>>> > - downloaded the src tarball and built the source (-Pdist -Pnative)
>>> > - verified the checksum
>>> > - brought up a secure pseudo distributed cluster
>>> > - did some basic file system operations (mkdir, list, put, cat) and
>>> > confirmed that everything was working
>>> > - confirmed that the web UI worked
>>> >
>>> > Best,
>>> > Aaron
>>> >
>>> > On Fri, Dec 8, 2017 at 12:31 PM, Andrew Wang <
>>> andrew.w...@cloudera.com>
>>> > wrote:
>>> >
>>> > > Hi all,
>>> > >
>>> > > Let me start, as always, by thanking the efforts of all the
>>> contributors
>>> > > who contributed to this release, especially those who jumped on
>>> the
>>> > issues
>>> > > found in RC0.
>>> > >
>>> > > I've prepared RC1 for Apache Hadoop 3.0.0. This release
>>> incorporates 302
>>> > > fixed JIRAs since the previous 3.0.0-beta1 release.
>>> > >
>>> > > You can find the artifacts here:
>>> > >
>>> > > http://home.apache.org/~wang/3.0.0-RC1/
>>> > >
>>> > > I've done the traditional testing of building from the source
>>> tarball and
>>> > > running a Pi job on a single node cluster. I also verified that
>>> the
>>> > shaded
>>> > > jars are not empty.
>>> > >
>>> > > Found one issue that create-release (probably due to the mvn
>>> deploy
>>> > change)
>>> > > didn't sign the artifacts, but I fixed that by calling mvn one
>>> more time.
>>> > > Available here:
>>> > >
>>> > > https://repository.apache.org/content/repositories/orgapache
>>> hadoop-1075/
>>> > >
>>> > > This release will run the standard 5 days, closing on Dec 13th at
>>> 12:31pm
>>> > > Pacific. My +1 to start.
>>> > >
>>> > > Best,
>>> > > Andrew
>>> > >
>>> >
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> -
>>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>>
>>
>>
>>
>>



-- 
Lei (Eddy) Xu
Software Engineer, Cloudera

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Lei Xu]

+1 binding

Run the following steps:

* Check md5 of sources and package.
* Run a YARN + HDFS pseudo cluster.
* Run terasuite on YARN.
* Run HDFS CLIs (ls , rm , etc)


On Mon, Nov 20, 2017 at 12:58 PM, Vinod Kumar Vavilapalli
 wrote:
> Quick question.
>
> I used to be able (in 2.x line) to create dist tarballs (mvn clean install 
> -Pdist -Dtar -DskipTests -Dmaven.javadoc.skip=true) from the source being 
> voted on (hadoop-3.0.0-src.tar.gz).
>
> The idea is to install HDFS, YARN, MR separately in separate root-directories 
> from the generated individual dist tarballs.
>
> But now I see that HDFS and common dist tarballs are empty
> -rw-r--r--  1 vinodkv  staff 45 Nov 20 12:39 
> ./hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0.tar.gz -
> -rw-r--r--  1 vinodkv  staff 45 Nov 20 12:40 
> ./hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-3.0.0.tar.gz
>
> But YARN and MR are fine
> -rw-r--r--  1 vinodkv  staff   64474187 Nov 20 12:41 
> ./hadoop-yarn-project/target/hadoop-yarn-project-3.0.0.tar.gz
> -rw-r--r--  1 vinodkv  staff   21674457 Nov 20 12:41 
> ./hadoop-mapreduce-project/target/hadoop-mapreduce-3.0.0.tar.gz
>
> Is it just me? Or is this broken?
>
> Thanks
> +Vinod
>
>> On Nov 14, 2017, at 1:34 PM, Andrew Wang  wrote:
>>
>> Hi folks,
>>
>> Thanks as always to the many, many contributors who helped with this
>> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
>> available here:
>>
>> http://people.apache.org/~wang/3.0.0-RC0/
>>
>> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>>
>> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
>> additions include the merge of YARN resource types, API-based configuration
>> of the CapacityScheduler, and HDFS router-based federation.
>>
>> I've done my traditional testing with a pseudo cluster and a Pi job. My +1
>> to start.
>>
>> Best,
>> Andrew
>
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>



-- 
Lei (Eddy) Xu
Software Engineer, Cloudera

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: 答复: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

[Lei Xu]

Hey,  Weiwei and Jitendra

Thanks a lot for this large effort to bring us ozone.

* As the current state of Ozone implementation, what are the major
benefits of using today’s Ozone over HDFS?  Giving that its missing
features like HDFS-12680 and HDFS-12697, being disabled by default,
and the closing of Hadoop 3.0 release, should we wait for a late merge
when Ozone is more mature ? Or more generally, why should this merge
to a release branch happen now, when Ozone is not yet usable by users?
Staying on a feature branch seems like it's still the right place to
me.
* For the existing HDFS user, could you address the semantic gaps
between Ozone / Ozone File System and HDFS. It would be great to
illustrate what is the expected use cases for Ozone giving its
different architecture and design decisions?  Like no append, no
atomic rename and etc.
* A follow question, was it able to run any of today’s Hadoop
applications (MR, Spark, Impala, Presto and etc) on Ozone directly, or
against OZoneFileSystem? I think a performance / scalability gain or
extended functionality should be the prerequisites for the merge.
Additionally, I believe such tests will reveal the potential caveats
if any.
* Ozone’s architecture shows great potential to address NN
scalability.  However it looks like a XXL effort to me, considering
the fact that 1) the community had multiple unfinished attempts to
simply separate namespace and block management within the same NN
process, and 2) many existing features like snapshot, append, erasure
coding, and etc, are not straightforward to be implemented in today’s
ozone design. Could you share your opinions on this matter?
* How stable is the ozone client? Should we mark them as unstable for
now? Also giving the significant difference between OzoneClient and
HdfsClient, should move it to a separated package or even a project? I
second Konstantin’s option to separate ozone from HDFS.
* Please add sections to the end-user and system admin oriented
documents for deploying and operating SCM, KSM, and also the chunk
servers on DataNodes. Additionally, the introduction in
“OZoneGettingStarted.md” is still building ozone from feature branch
HDFS-7240.

Best regards,

On Mon, Oct 23, 2017 at 11:10 AM, Jitendra Pandey
 wrote:
> I have filed https://issues.apache.org/jira/browse/HDFS-12697 to ensure ozone 
> stays disabled in a secure environment.
> Since ozone is disabled by default and will not come with security on, it 
> will not expose any new attack surface in a Hadoop deployment.
> Ozone security effort will need a detailed design and discussion on a 
> community jira. Hopefully, that effort will start soon after the merge.
>
> Thanks
> jitendra
>
> On 10/20/17, 2:40 PM, "larry mccay"  wrote:
>
> All -
>
> I broke this list of questions out into a separate DISCUSS thread where we
> can iterate over how a security audit process at merge time might look and
> whether it is even something that we want to take on.
>
> I will try and continue discussion on that thread and drive that to some
> conclusion before bringing it into any particular merge discussion.
>
> thanks,
>
> --larry
>
> On Fri, Oct 20, 2017 at 12:37 PM, larry mccay  wrote:
>
> > I previously sent this same email from my work email and it doesn't seem
> > to have gone through - resending from apache account (apologizing up 
> from
> > for the length)
> >
> > For such sizable merges in Hadoop, I would like to start doing security
> > audits in order to have an initial idea of the attack surface, the
> > protections available for known threats, what sort of configuration is
> > being used to launch processes, etc.
> >
> > I dug into the architecture documents while in the middle of this list -
> > nice docs!
> > I do intend to try and make a generic check list like this for such
> > security audits in the future so a lot of this is from that but I tried 
> to
> > also direct specific questions from those docs as well.
> >
> > 1. UIs
> > I see there are at least two UIs - Storage Container Manager and Key 
> Space
> > Manager. There are a number of typical vulnerabilities that we find in 
> UIs
> >
> > 1.1. What sort of validation is being done on any accepted user input?
> > (pointers to code would be appreciated)
> > 1.2. What explicit protections have been built in for (pointers to code
> > would be appreciated):
> >   1.2.1. cross site scripting
> >   1.2.2. cross site request forgery
> >   1.2.3. click jacking (X-Frame-Options)
> > 1.3. What sort of authentication is required for access to the UIs?
> > 1.4. What authorization is available for determining who can access what
> > capabilities of the UIs for either viewing, modifying data or affecting
> > object stores and related processes?
> > 1.5. Are the UIs built 

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha4-RC0

[Lei Xu]

+1 (binding)

Ran the following tests:
* Deploy a pesudo cluster using tar ball, run pi.
* Verified MD5 of tar balls for both src and dist.
* Build src tarball with -Pdist,tar

Thanks Andrew for the efforts!

On Thu, Jul 6, 2017 at 3:44 PM, Andrew Wang  wrote:
> Thanks all for the votes so far!
>
> I think we're still at a single binding +1 from myself, so I'll leave this
> vote open until we reach the minimum threshold of 3. I'm still hoping to
> can push the release out before the weekend.
>
> On Thu, Jul 6, 2017 at 2:58 PM, Vijaya Krishna Kalluru Subbarao <
> vij...@cloudera.com> wrote:
>
>> Ran Smokes and BVTs covering basic sanity testing(10+ tests ran) for all
>> these components:
>>
>>- Mapreduce(compression, archives, pipes, JHS),
>>- Avro(AvroMapreduce, HadoopAvro, HiveAvro, SqoopAvro),
>>- HBase(Balancer, compression, ImportExport, Snapshots, Schema
>>change),
>>- Oozie(Hive, Pig, Spark),
>>- Pig(PigAvro, PigParquet, PigCompression),
>>- Search(SolrCtlBasic, SolrRequestForwading, SolrSSLConfiguration).
>>
>> +1 non-binding.
>>
>> Regards,
>> Vijay
>>
>> On Thu, Jul 6, 2017 at 2:39 PM, Eric Badger > > wrote:
>>
>>> - Verified all checksums signatures
>>> - Built from src on macOS 10.12.5 with Java 1.8.0u65
>>> - Deployed single node pseudo cluster
>>> - Successfully ran sleep and pi jobs
>>> - Navigated the various UIs
>>>
>>> +1 (non-binding)
>>>
>>> Thanks,
>>>
>>> Eric
>>>
>>> On Thursday, July 6, 2017 3:31 PM, Aaron Fabbri 
>>> wrote:
>>>
>>>
>>>
>>> Thanks for the hard work on this!  +1 (non-binding)
>>>
>>> - Built from source tarball on OS X w/ Java 1.8.0_45.
>>> - Deployed mini/pseudo cluster.
>>> - Ran grep and wordcount examples.
>>> - Poked around ResourceManager and JobHistory UIs.
>>> - Ran all s3a integration tests in US West 2.
>>>
>>>
>>>
>>> On Thu, Jul 6, 2017 at 10:20 AM, Xiao Chen  wrote:
>>>
>>> > Thanks Andrew!
>>> > +1 (non-binding)
>>> >
>>> >- Verified md5's, checked tarball sizes are reasonable
>>> >- Built source tarball and deployed a pseudo-distributed cluster with
>>> >hdfs/kms
>>> >- Tested basic hdfs/kms operations
>>> >- Sanity checked webuis/logs
>>> >
>>> >
>>> > -Xiao
>>> >
>>> > On Wed, Jul 5, 2017 at 10:33 PM, John Zhuge 
>>> wrote:
>>> >
>>> > > +1 (non-binding)
>>> > >
>>> > >
>>> > >- Verified checksums and signatures of the tarballs
>>> > >- Built source with native, Java 1.8.0_131 on Mac OS X 10.12.5
>>> > >- Cloud connectors:
>>> > >   - A few S3A integration tests
>>> > >   - A few ADL live unit tests
>>> > >- Deployed both binary and built source to a pseudo cluster, passed
>>> > the
>>> > >following sanity tests in insecure, SSL, and SSL+Kerberos mode:
>>> > >   - HDFS basic and ACL
>>> > >   - DistCp basic
>>> > >   - WordCount (skipped in Kerberos mode)
>>> > >   - KMS and HttpFS basic
>>> > >
>>> > > Thanks Andrew for the great effort!
>>> > >
>>> > > On Wed, Jul 5, 2017 at 1:33 PM, Eric Payne >> > > invalid>
>>> > > wrote:
>>> > >
>>> > > > Thanks Andrew.
>>> > > > I downloaded the source, built it, and installed it onto a pseudo
>>> > > > distributed 4-node cluster.
>>> > > >
>>> > > > I ran mapred and streaming test cases, including sleep and
>>> wordcount.
>>> > > > +1 (non-binding)
>>> > > > -Eric
>>> > > >
>>> > > >   From: Andrew Wang 
>>> > > >  To: "common-...@hadoop.apache.org" ;
>>> "
>>> > > > hdfs-dev@hadoop.apache.org" ; "
>>> > > > mapreduce-...@hadoop.apache.org" ;
>>> "
>>> > > > yarn-...@hadoop.apache.org" 
>>> > > >  Sent: Thursday, June 29, 2017 9:41 PM
>>> > > >  Subject: [VOTE] Release Apache Hadoop 3.0.0-alpha4-RC0
>>> > > >
>>> > > > Hi all,
>>> > > >
>>> > > > As always, thanks to the many, many contributors who helped with
>>> this
>>> > > > release! I've prepared an RC0 for 3.0.0-alpha4:
>>> > > >
>>> > > > http://home.apache.org/~wang/3.0.0-alpha4-RC0/
>>> > > >
>>> > > > The standard 5-day vote would run until midnight on Tuesday, July
>>> 4th.
>>> > > > Given that July 4th is a holiday in the US, I expect this vote might
>>> > have
>>> > > > to be extended, but I'd like to close the vote relatively soon
>>> after.
>>> > > >
>>> > > > I've done my traditional testing of a pseudo-distributed cluster
>>> with a
>>> > > > single task pi job, which was successful.
>>> > > >
>>> > > > Normally my testing would end there, but I'm slightly more confident
>>> > this
>>> > > > time. At Cloudera, we've successfully packaged and deployed a
>>> snapshot
>>> > > from
>>> > > > a few days ago, and run basic smoke tests. Some bugs found from this
>>> > > > include HDFS-11956, which fixes backwards compat with Hadoop 2
>>> clients,
>>> > > and
>>> > > > 

Re: [VOTE] Release Apache Hadoop 2.6.5 (RC0)

[Lei Xu]

+1

The steps I've done:

* Downloaded release tar and source tar, verified MD5.
* Run a HDFS cluster, and copy files between local filesystem and HDFS.


On Tue, Sep 27, 2016 at 1:28 PM, Sangjin Lee  wrote:
> Hi folks,
>
> I have created a release candidate RC0 for the Apache Hadoop 2.6.5 release
> (the next maintenance release in the 2.6.x release line). Below are the
> details of this release candidate:
>
> The RC is available for validation at:
> http://home.apache.org/~sjlee/hadoop-2.6.5-RC0/.
>
> The RC tag in git is release-2.6.5-RC0 and its git commit is
> 6939fc935fba5651fdb33386d88aeb8e875cf27a.
>
> The maven artifacts are staged via repository.apache.org at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1048/.
>
> You can find my public key at
> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS.
>
> Please try the release and vote. The vote will run for the usual 5 days.
> Huge thanks to Chris Trezzo for spearheading the release management and
> doing all the work!
>
> Thanks,
> Sangjin



-- 
Lei (Eddy) Xu
Software Engineer, Cloudera

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] HDFS-1312 - Diskbalancer branch merge

[Lei Xu]

Thanks for the great work, Anu.

I have ran a test on a local cluster.  One suggestion is that the CLI
help messages should be more consistent with the rest of rest of hdfs
command line tools.

+1 for the merge.

On Wed, Jun 22, 2016 at 2:30 PM, Jitendra Pandey
 wrote:
> I think this is a very useful feature.
> I have file an improvement request, but that doesn’t block the merge. I think 
> the core functionality is complete, with good test coverage and documentation.
>
> +1 for the merge.
>
> On Jun 22, 2016, at 12:28 AM, Arpit Agarwal  wrote:
>
>> This feature is frequently requested by users and would be a useful addition 
>> to HDFS.
>>
>> I’ve code reviewed many of the sub-tasks and have tried out Disk Balancer in 
>> a test cluster. Suggested a couple of usability improvements. These are 
>> tracked by open Jiras but they need not hold up the merge. The documentation 
>> looks great.
>>
>> +1 for merging with HDFS-10557 fixed.
>>
>>
>> On 6/15/16, 5:38 PM, "Anu Engineer"  wrote:
>>
>> Hi All,
>>
>> I would like to propose a merge vote for HDFS-1312 (Disk balancer) branch to 
>> trunk. This branch creates a new tool that allows balancing of data on a 
>> datanode.
>>
>> The voting commences now and will run for 7 days till Jun/22/2016 5:00 PM 
>> PST.
>>
>> This tool distributes data evenly between the disks of same type on a 
>> datanode.
>> This is useful if a disk has been replaced or if some disks are out of space 
>> compared to rest of the disks.
>>
>> The current set of commands supported are:
>>
>> 1. Plan - Allows user to create a plan and review it. The plan describes how 
>> the data will be moved in the data node.
>>
>> 2. Execute - Allows execution of a plan against a datanode.
>>
>> 3. Query – Queries the status of disk balancer execution.
>>
>> 4. Cancel - cancels a running disk balancer plan.
>>
>> 5. Report – Reports the current state of data distribution on a node.
>>
>>
>> · The original proposal that captures the rationale and possible 
>> solution is here.  [ 
>> https://issues.apache.org/jira/secure/attachment/12755226/disk-balancer-proposal.pdf
>>  ]
>>
>> · The updated architecture and test plan document is here. [ 
>> https://issues.apache.org/jira/secure/attachment/12810720/Architecture_and_test_update.pdf
>>  ]
>>
>> · The merge patch that is a diff against trunk is posted here. [ 
>> https://issues.apache.org/jira/secure/attachment/12810943/HDFS-1312.001.patch
>>  ]
>>
>> · The user documentation which will be part of apache is posted 
>> here. [ 
>> https://issues.apache.org/jira/secure/attachment/12805976/HDFS-9547-HDFS-1312.002.patch
>>  ]
>>
>>
>> HDFS-1312 has a set of sub-tasks and they are ordered in the same sequence 
>> as they were committed to HDFS-1312. Hopefully this will make it easy to 
>> code review this branch.
>>
>> There are a set of commands which we would like to do later, including 
>> discovering which datanodes in the cluster would benefit by running disk 
>> balancer.
>> Appropriate JIRAs for these future work items are filed under HDFS-1312.
>>
>> Disk Balancer is made possible due to the work of many community members 
>> including Arpit Agarwal, Vinayakumar B, Mingliang Liu, Tsz Wo Nicholas Sze,
>> Lei (Eddy) Xu and Xiaobing Zhou. I would like to thank them all for the 
>> effort and support.
>>
>> Thanks
>> Anu
>>
>>
>>
>
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>



-- 
Lei (Eddy) Xu
Software Engineer, Cloudera

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.1 RC0

[Lei Xu]

+1 binding

Downloaded src and bin distribution, verified md5, sha1 and sha256
checksums of both tar files.
Built src using mvn package.
Ran a pseudo HDFS cluster
Ran dfs -put some files, and checked files on NN's web interface.



On Mon, Jun 29, 2015 at 11:54 AM, Wangda Tan wheele...@gmail.com wrote:
 +1 (non-binding)

 Compiled and deployed a single node cluster, tried to change node labels
 and run distributed_shell with node label specified.

 On Mon, Jun 29, 2015 at 10:30 AM, Ted Yu yuzhih...@gmail.com wrote:

 +1 (non-binding)

 Compiled hbase branch-1 with Java 1.8.0_45
 Ran unit test suite which passed.

 On Mon, Jun 29, 2015 at 7:22 AM, Steve Loughran ste...@hortonworks.com
 wrote:

 
  +1 binding from me.
 
  Tests:
 
  Rebuild slider with Hadoop.version=2.7.1; ran all the tests including
  against a secure cluster.
  Repeated for windows running Java 8.
 
  All tests passed
 
 
   On 29 Jun 2015, at 09:45, Vinod Kumar Vavilapalli vino...@apache.org
  wrote:
  
   Hi all,
  
   I've created a release candidate RC0 for Apache Hadoop 2.7.1.
  
   As discussed before, this is the next stable release to follow up
 2.6.0,
   and the first stable one in the 2.7.x line.
  
   The RC is available for validation at:
   *http://people.apache.org/~vinodkv/hadoop-2.7.1-RC0/
   http://people.apache.org/~vinodkv/hadoop-2.7.1-RC0/*
  
   The RC tag in git is: release-2.7.1-RC0
  
   The maven artifacts are available via repository.apache.org at
   *
  https://repository.apache.org/content/repositories/orgapachehadoop-1019/
   
  https://repository.apache.org/content/repositories/orgapachehadoop-1019/
 *
  
   Please try the release and vote; the vote will run for the usual 5
 days.
  
   Thanks,
   Vinod
  
   PS: It took 2 months instead of the planned [1] 2 weeks in getting this
   release out: post-mortem in a separate thread.
  
   [1]: A 2.7.1 release to follow up 2.7.0
   http://markmail.org/thread/zwzze6cqqgwq4rmw
 
 




-- 
Lei (Eddy) Xu
Software Engineer, Cloudera

Re: upstream jenkins build broken?

[Lei Xu]

I filed HDFS-7917 to change the way to simulate disk failures.

But I think we still need infrastructure folks to help with jenkins
scripts to clean the dirs left today.

On Fri, Mar 13, 2015 at 1:38 PM, Mai Haohui ricet...@gmail.com wrote:
 Any updates on this issues? It seems that all HDFS jenkins builds are
 still failing.

 Regards,
 Haohui

 On Thu, Mar 12, 2015 at 12:53 AM, Vinayakumar B vinayakum...@apache.org 
 wrote:
 I think the problem started from here.

 https://builds.apache.org/job/PreCommit-HDFS-Build/9828/testReport/junit/org.apache.hadoop.hdfs.server.datanode/TestDataNodeVolumeFailure/testUnderReplicationAfterVolFailure/

 As Chris mentioned TestDataNodeVolumeFailure is changing the permission.
 But in this patch, ReplicationMonitor got NPE and it got terminate signal,
 due to which MiniDFSCluster.shutdown() throwing Exception.

 But, TestDataNodeVolumeFailure#teardown() is restoring those permission
 after shutting down cluster. So in this case IMO, permissions were never
 restored.


   @After
   public void tearDown() throws Exception {
 if(data_fail != null) {
   FileUtil.setWritable(data_fail, true);
 }
 if(failedDir != null) {
   FileUtil.setWritable(failedDir, true);
 }
 if(cluster != null) {
   cluster.shutdown();
 }
 for (int i = 0; i  3; i++) {
   FileUtil.setExecutable(new File(dataDir, data+(2*i+1)), true);
   FileUtil.setExecutable(new File(dataDir, data+(2*i+2)), true);
 }
   }


 Regards,
 Vinay

 On Thu, Mar 12, 2015 at 12:35 PM, Vinayakumar B vinayakum...@apache.org
 wrote:

 When I see the history of these kind of builds, All these are failed on
 node H9.

 I think some or the other uncommitted patch would have created the problem
 and left it there.


 Regards,
 Vinay

 On Thu, Mar 12, 2015 at 6:16 AM, Sean Busbey bus...@cloudera.com wrote:

 You could rely on a destructive git clean call instead of maven to do the
 directory removal.

 --
 Sean
 On Mar 11, 2015 4:11 PM, Colin McCabe cmcc...@alumni.cmu.edu wrote:

  Is there a maven plugin or setting we can use to simply remove
  directories that have no executable permissions on them?  Clearly we
  have the permission to do this from a technical point of view (since
  we created the directories as the jenkins user), it's simply that the
  code refuses to do it.
 
  Otherwise I guess we can just fix those tests...
 
  Colin
 
  On Tue, Mar 10, 2015 at 2:43 PM, Lei Xu l...@cloudera.com wrote:
   Thanks a lot for looking into HDFS-7722, Chris.
  
   In HDFS-7722:
   TestDataNodeVolumeFailureXXX tests reset data dir permissions in
  TearDown().
   TestDataNodeHotSwapVolumes reset permissions in a finally clause.
  
   Also I ran mvn test several times on my machine and all tests passed.
  
   However, since in DiskChecker#checkDirAccess():
  
   private static void checkDirAccess(File dir) throws
 DiskErrorException {
 if (!dir.isDirectory()) {
   throw new DiskErrorException(Not a directory: 
+ dir.toString());
 }
  
 checkAccessByFileMethods(dir);
   }
  
   One potentially safer alternative is replacing data dir with a regular
   file to stimulate disk failures.
  
   On Tue, Mar 10, 2015 at 2:19 PM, Chris Nauroth 
 cnaur...@hortonworks.com
  wrote:
   TestDataNodeHotSwapVolumes, TestDataNodeVolumeFailure,
   TestDataNodeVolumeFailureReporting, and
   TestDataNodeVolumeFailureToleration all remove executable permissions
  from
   directories like the one Colin mentioned to simulate disk failures at
  data
   nodes.  I reviewed the code for all of those, and they all appear to
 be
   doing the necessary work to restore executable permissions at the
 end of
   the test.  The only recent uncommitted patch I¹ve seen that makes
  changes
   in these test suites is HDFS-7722.  That patch still looks fine
  though.  I
   don¹t know if there are other uncommitted patches that changed these
  test
   suites.
  
   I suppose it¹s also possible that the JUnit process unexpectedly died
   after removing executable permissions but before restoring them.
 That
   always would have been a weakness of these test suites, regardless of
  any
   recent changes.
  
   Chris Nauroth
   Hortonworks
   http://hortonworks.com/
  
  
  
  
  
  
   On 3/10/15, 1:47 PM, Aaron T. Myers a...@cloudera.com wrote:
  
  Hey Colin,
  
  I asked Andrew Bayer, who works with Apache Infra, what's going on
 with
  these boxes. He took a look and concluded that some perms are being
 set
  in
  those directories by our unit tests which are precluding those files
  from
  getting deleted. He's going to clean up the boxes for us, but we
 should
  expect this to keep happening until we can fix the test in question
 to
  properly clean up after itself.
  
  To help narrow down which commit it was that started this, Andrew
 sent
  me
  this info:
  
  /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-
 
 Build/hadoop-hdfs-project/hadoop-hdfs/target/test

Re: upstream jenkins build broken?

[Lei Xu]

Thanks a lot for looking into HDFS-7722, Chris.

In HDFS-7722:
TestDataNodeVolumeFailureXXX tests reset data dir permissions in TearDown().
TestDataNodeHotSwapVolumes reset permissions in a finally clause.

Also I ran mvn test several times on my machine and all tests passed.

However, since in DiskChecker#checkDirAccess():

private static void checkDirAccess(File dir) throws DiskErrorException {
  if (!dir.isDirectory()) {
throw new DiskErrorException(Not a directory: 
 + dir.toString());
  }

  checkAccessByFileMethods(dir);
}

One potentially safer alternative is replacing data dir with a regular
file to stimulate disk failures.

On Tue, Mar 10, 2015 at 2:19 PM, Chris Nauroth cnaur...@hortonworks.com wrote:
 TestDataNodeHotSwapVolumes, TestDataNodeVolumeFailure,
 TestDataNodeVolumeFailureReporting, and
 TestDataNodeVolumeFailureToleration all remove executable permissions from
 directories like the one Colin mentioned to simulate disk failures at data
 nodes.  I reviewed the code for all of those, and they all appear to be
 doing the necessary work to restore executable permissions at the end of
 the test.  The only recent uncommitted patch I¹ve seen that makes changes
 in these test suites is HDFS-7722.  That patch still looks fine though.  I
 don¹t know if there are other uncommitted patches that changed these test
 suites.

 I suppose it¹s also possible that the JUnit process unexpectedly died
 after removing executable permissions but before restoring them.  That
 always would have been a weakness of these test suites, regardless of any
 recent changes.

 Chris Nauroth
 Hortonworks
 http://hortonworks.com/






 On 3/10/15, 1:47 PM, Aaron T. Myers a...@cloudera.com wrote:

Hey Colin,

I asked Andrew Bayer, who works with Apache Infra, what's going on with
these boxes. He took a look and concluded that some perms are being set in
those directories by our unit tests which are precluding those files from
getting deleted. He's going to clean up the boxes for us, but we should
expect this to keep happening until we can fix the test in question to
properly clean up after itself.

To help narrow down which commit it was that started this, Andrew sent me
this info:

/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-
Build/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data3/ has
500 perms, so I'm guessing that's the problem. Been that way since 9:32
UTC
on March 5th.

--
Aaron T. Myers
Software Engineer, Cloudera

On Tue, Mar 10, 2015 at 1:24 PM, Colin P. McCabe cmcc...@apache.org
wrote:

 Hi all,

 A very quick (and not thorough) survey shows that I can't find any
 jenkins jobs that succeeded from the last 24 hours.  Most of them seem
 to be failing with some variant of this message:

 [ERROR] Failed to execute goal
 org.apache.maven.plugins:maven-clean-plugin:2.5:clean (default-clean)
 on project hadoop-hdfs: Failed to clean project: Failed to delete


/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/hadoop-hdfs-pr
oject/hadoop-hdfs/target/test/data/dfs/data/data3
 - [Help 1]

 Any ideas how this happened?  Bad disk, unit test setting wrong
 permissions?

 Colin





-- 
Lei (Eddy) Xu
Software Engineer, Cloudera

Re: Looking to a Hadoop 3 release

[Lei Xu]

+1.  Would love to help.



On Mon, Mar 2, 2015 at 3:19 PM, Andrew Wang andrew.w...@cloudera.com wrote:
 Hi devs,

 It's been a year and a half since 2.x went GA, and I think we're about due
 for a 3.x release.
 Notably, there are two incompatible changes I'd like to call out, that will
 have a tremendous positive impact for our users.

 First, classpath isolation being done at HADOOP-11656, which has been a
 long-standing request from many downstreams and Hadoop users.

 Second, bumping the source and target JDK version to JDK8 (related to
 HADOOP-11090), which is important since JDK7 is EOL in April 2015 (two
 months from now). In the past, we've had issues with our dependencies
 discontinuing support for old JDKs, so this will future-proof us.

 Between the two, we'll also have quite an opportunity to clean up and
 upgrade our dependencies, another common user and developer request.

 I'd like to propose that we start rolling a series of monthly-ish series of
 3.0 alpha releases ASAP, with myself volunteering to take on the RM and
 other cat herding responsibilities. There are already quite a few changes
 slated for 3.0 besides the above (for instance the shell script rewrite) so
 there's already value in a 3.0 alpha, and the more time we give downstreams
 to integrate, the better.

 This opens up discussion about inclusion of other changes, but I'm hoping
 to freeze incompatible changes after maybe two alphas, do a beta (with no
 further incompat changes allowed), and then finally a 3.x GA. For those
 keeping track, that means a 3.x GA in about four months.

 I would also like to stress though that this is not intended to be a big
 bang release. For instance, it would be great if we could maintain wire
 compatibility between 2.x and 3.x, so rolling upgrades work. Keeping
 branch-2 and branch-3 similar also makes backports easier, since we're
 likely maintaining 2.x for a while yet.

 Please let me know any comments / concerns related to the above. If people
 are friendly to the idea, I'd like to cut a branch-3 and start working on
 the first alpha.

 Best,
 Andrew



-- 
Lei (Eddy) Xu
Software Engineer, Cloudera

[jira] [Created] (HDFS-6521) Improve the readability of 'hadoop fs -help'

[Lei Xu (JIRA)]

Lei Xu created HDFS-6521:


 Summary: Improve the readability of 'hadoop fs -help'
 Key: HDFS-6521
 URL: https://issues.apache.org/jira/browse/HDFS-6521
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: tools
Affects Versions: 2.5.0
Reporter: Lei Xu
Assignee: Lei Xu
Priority: Minor
 Fix For: 2.5.0


'hadoop fs -help` displays help informations with numbers of different formats. 

This patch borrows the format used in `hdfs cacheadmin -help`: all options are 
formatted by using org.apache.hadoop.tools.TableListing.



--
This message was sent by Atlassian JIRA
(v6.2#6252)