Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2

[Arpit Agarwal]

Hi Duo,

Thank you for starting this discussion. Log4j1.2 bridge seems like a practical 
short-term solution. However the bridge will silently affect applications that 
add appenders or filters. NameNode audit logger and metrics come to mind. There 
may be others.

Thanks,
Arpit


> On Jan 20, 2022, at 5:55 AM, Duo Zhang  wrote:
> 
> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think it
> is time to speed up the migration to log4j2 work[4] now.
> 
> You can see the discussion on the jira issue[4], our goal is to fully
> migrate to log4j2 and the current most blocking issue is lack of the
> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already
> started a discussion thread on the log4j dev mailing list[5] and the result
> is optimistic and I've filed an issue for log4j2[6], but I do not think it
> could be addressed and released soon. If we want to fully migrate to
> log4j2, then either we introduce new environment variables or split the old
> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the
> complexity of our current startup scripts, the work is not easy and it will
> also break lots of other hadoop deployment systems if they do not use our
> startup scripts...
> 
> So after reconsidering the current situation, I prefer we use the log4j1.2
> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
> addressed and released, we start to fully migrate to log4j2. Of course we
> have other problems for log4j1.2 bridge too, as we have TaskLogAppender,
> ContainerLogAppender and ContainerRollingLogAppender which inherit
> FileAppender and RollingFileAppender in log4j1, which are not part of the
> log4j1.2 bridge. But anyway, at least we could just copy the source code to
> hadoop as we have WriteAppender in log4j1.2 bridge, and these two classes
> do not have related CVEs.
> 
> Thoughts? For me I would like us to make a new 3.4.x release line to remove
> the log4j1 dependencies ASAP.
> 
> Thanks.
> 
> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
> 4. https://issues.apache.org/jira/browse/HADOOP-16206
> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
> 6. https://issues.apache.org/jira/browse/LOG4J2-3341


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] which release lines should we still consider actively maintained?

[Arpit Agarwal]

+1 to EOL 3.1.x at least.


> On May 23, 2021, at 9:51 PM, Wei-Chiu Chuang  
> wrote:
> 
> Sean,
> 
> For reasons I don't understand, I never received emails from your new
> address in the mailing list. Only Akira's response.
> 
> I was just able to start a thread like this.
> 
> I am +1 to EOL 3.1.5.
> Reason? Spark is already on Hadoop 3.2. Hive and Tez are actively working
> to support Hadoop 3.3. HBase supports Hadoop 3.3 already. They are the most
> common Hadoop applications so I think a 3.1 isn't that necessarily
> important.
> 
> With Hadoop 3.3.1, we have a number of improvements to support a better
> HDFS upgrade experience, so upgrading from Hadoop 3.1 should be relatively
> easy. Application upgrade takes some effort though (commons-lang ->
> commons-lang3 migration for example)
> I've been maintaining the HDFS code in branch-3.1, so from a
> HDFS perspective the branch is always in a ready to release state.
> 
> The Hadoop 3.1 line is more than 3 years old. Maintaining this branch is
> getting trickier. I am +100 to reduce the number of actively maintained
> release line. IMO, 2 Hadoop 3 lines + 1 Hadoop 2 line is a good idea.
> 
> 
> 
> For Hadoop 3.3 line: If no one beats me, I plan to make a 3.3.2 in 2-3
> months. And another one in another 2-3 months.
> The Hadoop 3.3.1 has nearly 700 commits not in 3.3.0. It is very difficult
> to make/validate a maint release with such a big divergence in the code.
> 
> 
> On Mon, May 24, 2021 at 12:06 PM Akira Ajisaka  > wrote:
> 
>> Hi Sean,
>> 
>> Thank you for starting the discussion.
>> 
>> I think branch-2.10, branch-3.1, branch-3.2, branch-3.3, and trunk
>> (3.4.x) are actively maintained.
>> 
>> The next releases will be:
>> - 3.4.0
>> - 3.3.1 (Thanks, Wei-Chiu!)
>> - 3.2.3
>> - 3.1.5
>> - 2.10.2
>> 
>>> Are there folks willing to go through being release managers to get more
>> of these release lines on a steady cadence?
>> 
>> Now I'm interested in becoming a release manager of 3.1.5.
>> 
>>> If I were to take up maintenance release for one of them which should it
>> be?
>> 
>> 3.2.3 or 2.10.2 seems to be a good choice.
>> 
>>> Should we declare to our downstream users that some of these lines
>> aren’t going to get more releases?
>> 
>> Now I think we don't need to declare that. I believe 3.3.1, 3.2.3,
>> 3.1.5, and 2.10.2 will be released in the near future.
>> There are some earlier discussions of 3.1.x EoL, so 3.1.5 may be a
>> final release of the 3.1.x release line.
>> 
>>> Is there downstream facing documentation somewhere that I missed for
>> setting expectations about our release cadence and actively maintained
>> branches?
>> 
>> As you commented, the confluence wiki pages for Hadoop releases were
>> out of date. Updated [1].
>> 
>>> Do we have a backlog of work written up that could make the release
>> process easier for our release managers?
>> 
>> The release process is documented and maintained:
>> https://cwiki.apache.org/confluence/display/HADOOP2/HowToRelease
>> Also, there are some backlogs [1], [2].
>> 
>> [1]:
>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Active+Release+Lines
>> [2]: https://cwiki.apache.org/confluence/display/HADOOP/Roadmap
>> 
>> Thanks,
>> Akira
>> 
>> On Fri, May 21, 2021 at 7:12 AM Sean Busbey 
>> wrote:
>>> 
>>> 
>>> Hi folks!
>>> 
>>> Which release lines do we as a community still consider actively
>> maintained?
>>> 
>>> I found an earlier discussion[1] where we had consensus to consider
>> branches that don’t get maintenance releases on a regular basis end-of-life
>> for practical purposes. The result of that discussion was written up in our
>> wiki docs in the “EOL Release Branches” page, summarized here
>>> 
 If no volunteer to do a maintenance release in a short to mid-term
>> (like 3 months to 1 or 1.5 year).
>>> 
>>> Looking at release lines that are still on our download page[3]:
>>> 
>>> * Hadoop 2.10.z - last release 8 months ago
>>> * Hadoop 3.1.z - last release 9.5 months ago
>>> * Hadoop 3.2.z - last release 4.5 months ago
>>> * Hadoop 3.3.z - last release 10 months ago
>>> 
>>> And then trunk holds 3.4 which hasn’t had a release since the branch-3.3
>> fork ~14 months ago.
>>> 
>>> I can see that Wei-Chiu has been actively working on getting the 3.3.1
>> release out[4] (thanks Wei-Chiu!) but I do not see anything similar for the
>> other release lines.
>>> 
>>> We also have pages on the wiki for our project roadmap of release[5],
>> but it seems out of date since it lists in progress releases that have
>> happened or branches we have announced as end of life, i.e. 2.8.
>>> 
>>> We also have a group of pages (sorry, I’m not sure what the confluence
>> jargon is for this) for “hadoop active release lines”[6] but this list has
>> 2.8, 2.9, 3.0, 3.1, and 3.3. So several declared end of life lines and no
>> 2.10 or 3.2 despite those being our release lines with the most recent
>> releases.
>>> 
>>> Are there folks willing to go through being rele

Re: [VOTE] Apache Hadoop Ozone 0.5.0-beta RC2

[Arpit Agarwal]

+1 binding.

- Verified hashes and signatures
- Built from source
- Deployed to 5 node cluster
- Tried ozone shell and filesystem operations
- Ran freon stress test for a while with write validation


I couldn’t find the RC2 tag in the gitbox repo, although it is there in GitHub.

Thanks,
Arpit



> On Mar 21, 2020, at 3:57 PM, Hanisha Koneru  
> wrote:
> 
> Thank you Dinesh for putting up the RCs.
> 
> +1 binding.
> 
> Verified the following:
>  - Built from source
>  - Deployed to 5 node docker cluster and ran sanity tests.
>  - Ran smoke tests
> 
> Thanks
> Hanisha
> 
>> On Mar 15, 2020, at 7:27 PM, Dinesh Chitlangia  wrote:
>> 
>> Hi Folks,
>> 
>> We have put together RC2 for Apache Hadoop Ozone 0.5.0-beta.
>> 
>> The RC artifacts are at:
>> https://home.apache.org/~dineshc/ozone-0.5.0-rc2/
>> 
>> The public key used for signing the artifacts can be found at:
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> 
>> The maven artifacts are staged at:
>> https://repository.apache.org/content/repositories/orgapachehadoop-1262
>> 
>> The RC tag in git is at:
>> https://github.com/apache/hadoop-ozone/tree/ozone-0.5.0-beta-RC2
>> 
>> This release contains 800+ fixes/improvements [1].
>> Thanks to everyone who put in the effort to make this happen.
>> 
>> *The vote will run for 7 days, ending on March 22nd 2020 at 11:59 pm PST.*
>> 
>> Note: This release is beta quality, it’s not recommended to use in
>> production but we believe that it’s stable enough to try out the feature
>> set and collect feedback.
>> 
>> 
>> [1] https://s.apache.org/ozone-0.5.0-fixed-issues
>> 
>> Thanks,
>> Dinesh Chitlangia
> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary

[Arpit Agarwal]

Thanks for the clarification Brahma. Can you update the proposal to state that 
it is optional (it may help to put the proposal on cwiki)?

Also if we go ahead then the RM documentation should be clear this is an 
optional step.


> On Mar 17, 2020, at 11:06 AM, Brahma Reddy Battula  wrote:
> 
> Sure, we can't make mandatory while voting and we can upload to downloads
> once release vote is passed.
> 
> On Tue, 17 Mar 2020 at 11:24 PM, Arpit Agarwal
>  wrote:
> 
>>> Sorry,didn't get you...do you mean, once release voting is
>>> processed and upload by RM..?
>> 
>> Yes, that is what I meant. I don’t want us to make more mandatory work for
>> the release manager because the job is hard enough already.
>> 
>> 
>>> On Mar 17, 2020, at 10:46 AM, Brahma Reddy Battula 
>> wrote:
>>> 
>>> Sorry,didn't get you...do you mean, once release voting is processed and
>>> upload by RM..?
>>> 
>>> FYI. There is docker image for ARM also which support all scripts
>>> (createrelease, start-build-env.sh, etc ).
>>> 
>>> https://issues.apache.org/jira/browse/HADOOP-16797
>>> 
>>> On Tue, Mar 17, 2020 at 10:59 PM Arpit Agarwal
>>>  wrote:
>>> 
>>>> Can ARM binaries be provided after the fact? We cannot increase the RM’s
>>>> burden by asking them to generate an extra set of binaries.
>>>> 
>>>> 
>>>>> On Mar 17, 2020, at 10:23 AM, Brahma Reddy Battula 
>>>> wrote:
>>>>> 
>>>>> + Dev mailing list.
>>>>> 
>>>>> -- Forwarded message -
>>>>> From: Brahma Reddy Battula 
>>>>> Date: Tue, Mar 17, 2020 at 10:31 PM
>>>>> Subject: Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary
>>>>> To: junping_du 
>>>>> 
>>>>> 
>>>>> thanks junping for your reply.
>>>>> 
>>>>> bq.  I think most of us in Hadoop community doesn't want to have
>>>> biased
>>>>> on ARM or any other platforms.
>>>>> 
>>>>> Yes, release voting will be based on the source code.AFAIK,Binary we
>> are
>>>>> providing for user to easy to download and verify.
>>>>> 
>>>>> bq. The only thing I try to understand is how much complexity get
>>>>> involved for our RM work. Does that potentially become a blocker for
>>>> future
>>>>> releases? And how we can get rid of this risk.
>>>>> 
>>>>> As I mentioned earlier, RM need to access the ARM machine(it will be
>>>>> donated and current qbt also using one ARM machine) and build tar using
>>>> the
>>>>> keys. As it can be common machine, RM can delete his keys once release
>>>>> approved.
>>>>> Can be sorted out as I mentioned earlier.(For accessing the ARM
>> machine)
>>>>> 
>>>>> bq.   If you can list the concrete work that RM need to do extra
>> for
>>>>> ARM release, that would help us to better understand.
>>>>> 
>>>>> I can write and update for future reference.
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On Tue, Mar 17, 2020 at 10:41 AM 俊平堵  wrote:
>>>>> 
>>>>>> Hi Brahma,
>>>>>>   I think most of us in Hadoop community doesn't want to have biased
>>>> on
>>>>>> ARM or any other platforms.
>>>>>>   The only thing I try to understand is how much complexity get
>>>>>> involved for our RM work. Does that potentially become a blocker for
>>>> future
>>>>>> releases? And how we can get rid of this risk.
>>>>>>If you can list the concrete work that RM need to do extra for ARM
>>>>>> release, that would help us to better understand.
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>> Junping
>>>>>> 
>>>>>> Akira Ajisaka  于2020年3月13日周五 上午12:34写道：
>>>>>> 
>>>>>>> If you can provide ARM release for future releases, I'm fine with
>> that.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Akira
>>>>>>> 
>>>>>>> 

Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary

[Arpit Agarwal]

> Sorry,didn't get you...do you mean, once release voting is
> processed and upload by RM..?

Yes, that is what I meant. I don’t want us to make more mandatory work for the 
release manager because the job is hard enough already.


> On Mar 17, 2020, at 10:46 AM, Brahma Reddy Battula  wrote:
> 
> Sorry,didn't get you...do you mean, once release voting is processed and
> upload by RM..?
> 
> FYI. There is docker image for ARM also which support all scripts
> (createrelease, start-build-env.sh, etc ).
> 
> https://issues.apache.org/jira/browse/HADOOP-16797
> 
> On Tue, Mar 17, 2020 at 10:59 PM Arpit Agarwal
>  wrote:
> 
>> Can ARM binaries be provided after the fact? We cannot increase the RM’s
>> burden by asking them to generate an extra set of binaries.
>> 
>> 
>>> On Mar 17, 2020, at 10:23 AM, Brahma Reddy Battula 
>> wrote:
>>> 
>>> + Dev mailing list.
>>> 
>>> -- Forwarded message -
>>> From: Brahma Reddy Battula 
>>> Date: Tue, Mar 17, 2020 at 10:31 PM
>>> Subject: Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary
>>> To: junping_du 
>>> 
>>> 
>>> thanks junping for your reply.
>>> 
>>> bq.  I think most of us in Hadoop community doesn't want to have
>> biased
>>> on ARM or any other platforms.
>>> 
>>> Yes, release voting will be based on the source code.AFAIK,Binary we are
>>> providing for user to easy to download and verify.
>>> 
>>> bq. The only thing I try to understand is how much complexity get
>>> involved for our RM work. Does that potentially become a blocker for
>> future
>>> releases? And how we can get rid of this risk.
>>> 
>>> As I mentioned earlier, RM need to access the ARM machine(it will be
>>> donated and current qbt also using one ARM machine) and build tar using
>> the
>>> keys. As it can be common machine, RM can delete his keys once release
>>> approved.
>>> Can be sorted out as I mentioned earlier.(For accessing the ARM machine)
>>> 
>>> bq.   If you can list the concrete work that RM need to do extra for
>>> ARM release, that would help us to better understand.
>>> 
>>> I can write and update for future reference.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Tue, Mar 17, 2020 at 10:41 AM 俊平堵  wrote:
>>> 
>>>> Hi Brahma,
>>>>I think most of us in Hadoop community doesn't want to have biased
>> on
>>>> ARM or any other platforms.
>>>>The only thing I try to understand is how much complexity get
>>>> involved for our RM work. Does that potentially become a blocker for
>> future
>>>> releases? And how we can get rid of this risk.
>>>> If you can list the concrete work that RM need to do extra for ARM
>>>> release, that would help us to better understand.
>>>> 
>>>> Thanks,
>>>> 
>>>> Junping
>>>> 
>>>> Akira Ajisaka  于2020年3月13日周五 上午12:34写道：
>>>> 
>>>>> If you can provide ARM release for future releases, I'm fine with that.
>>>>> 
>>>>> Thanks,
>>>>> Akira
>>>>> 
>>>>> On Thu, Mar 12, 2020 at 9:41 PM Brahma Reddy Battula <
>> bra...@apache.org>
>>>>> wrote:
>>>>> 
>>>>>> thanks Akira.
>>>>>> 
>>>>>> Currently only problem is dedicated ARM for future RM.This i want to
>>>>> sort
>>>>>> out like below,if you've some other,please let me know.
>>>>>> 
>>>>>> i) Single machine and share cred to future RM ( as we can delete keys
>>>>> once
>>>>>> release is over).
>>>>>> ii) Creating the jenkins project ( may be we need to discuss in the
>>>>>> board..)
>>>>>> iii) I can provide ARM release for future releases.
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Thu, Mar 12, 2020 at 5:14 PM Akira Ajisaka 
>>>>> wrote:
>>>>>> 
>>>>>>> Hi Brahma,
>>>>>>> 
>>>>>>> I think we cannot do any of your proposed actions.
>>>>>>> 
>>>

Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary

[Arpit Agarwal]

Can ARM binaries be provided after the fact? We cannot increase the RM’s burden 
by asking them to generate an extra set of binaries.


> On Mar 17, 2020, at 10:23 AM, Brahma Reddy Battula  wrote:
> 
> + Dev mailing list.
> 
> -- Forwarded message -
> From: Brahma Reddy Battula 
> Date: Tue, Mar 17, 2020 at 10:31 PM
> Subject: Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary
> To: junping_du 
> 
> 
> thanks junping for your reply.
> 
> bq.  I think most of us in Hadoop community doesn't want to have biased
> on ARM or any other platforms.
> 
> Yes, release voting will be based on the source code.AFAIK,Binary we are
> providing for user to easy to download and verify.
> 
> bq. The only thing I try to understand is how much complexity get
> involved for our RM work. Does that potentially become a blocker for future
> releases? And how we can get rid of this risk.
> 
> As I mentioned earlier, RM need to access the ARM machine(it will be
> donated and current qbt also using one ARM machine) and build tar using the
> keys. As it can be common machine, RM can delete his keys once release
> approved.
> Can be sorted out as I mentioned earlier.(For accessing the ARM machine)
> 
> bq.   If you can list the concrete work that RM need to do extra for
> ARM release, that would help us to better understand.
> 
> I can write and update for future reference.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> On Tue, Mar 17, 2020 at 10:41 AM 俊平堵  wrote:
> 
>> Hi Brahma,
>> I think most of us in Hadoop community doesn't want to have biased on
>> ARM or any other platforms.
>> The only thing I try to understand is how much complexity get
>> involved for our RM work. Does that potentially become a blocker for future
>> releases? And how we can get rid of this risk.
>>  If you can list the concrete work that RM need to do extra for ARM
>> release, that would help us to better understand.
>> 
>> Thanks,
>> 
>> Junping
>> 
>> Akira Ajisaka  于2020年3月13日周五 上午12:34写道：
>> 
>>> If you can provide ARM release for future releases, I'm fine with that.
>>> 
>>> Thanks,
>>> Akira
>>> 
>>> On Thu, Mar 12, 2020 at 9:41 PM Brahma Reddy Battula 
>>> wrote:
>>> 
 thanks Akira.
 
 Currently only problem is dedicated ARM for future RM.This i want to
>>> sort
 out like below,if you've some other,please let me know.
 
 i) Single machine and share cred to future RM ( as we can delete keys
>>> once
 release is over).
 ii) Creating the jenkins project ( may be we need to discuss in the
 board..)
 iii) I can provide ARM release for future releases.
 
 
 
 
 
 
 
 On Thu, Mar 12, 2020 at 5:14 PM Akira Ajisaka 
>>> wrote:
 
> Hi Brahma,
> 
> I think we cannot do any of your proposed actions.
> 
> 
 
>>> http://www.apache.org/legal/release-policy.html#owned-controlled-hardware
>> Strictly speaking, releases must be verified on hardware owned and
> controlled by the committer. That means hardware the committer has
 physical
> possession and control of and exclusively full
>>> administrative/superuser
> access to. That's because only such hardware is qualified to hold a
>>> PGP
> private key, and the release should be verified on the machine the
 private
> key lives on or on a machine as trusted as that.
> 
> https://www.apache.org/dev/release-distribution.html#sigs-and-sums
>> Private keys MUST NOT be stored on any ASF machine. Likewise,
 signatures
> for releases MUST NOT be created on ASF machines.
> 
> We need to have dedicated physical ARM machines for each release
>>> manager,
> and now it is not feasible.
> If you provide an unofficial ARM binary release in some repository,
 that's
> okay.
> 
> -Akira
> 
> On Thu, Mar 12, 2020 at 7:57 PM Brahma Reddy Battula <
>>> bra...@apache.org>
> wrote:
> 
>> Hello folks,
>> 
>> As currently trunk will support ARM based compilation and qbt(1) is
>> running
>> from several months with quite stable, hence planning to propose ARM
>> binary
>> this time.
>> 
>> ( Note : As we'll know voting will be based on the source,so this
>>> will
 not
>> issue.)
>> 
>> *Proposed Change:*
>> Currently in downloads we are keeping only x86 binary(2),Can we keep
>>> ARM
>> binary also.?
>> 
>> *Actions:*
>> a) *Dedicated* *Machine*:
>>   i) Dedicated ARM machine will be donated which I confirmed
>>   ii) Or can use jenkins ARM machine itself which is currently
>>> used
>> for ARM
>> b) *Automate Release:* How about having one release project in
 jenkins..?
>> So that future RM's just trigger the jenkin project.
>> 
>> Please let me know your thoughts on this.
>> 
>> 
>> 1.
>> 
>> 
 
>>> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/
>> 2.http

Re: [VOTE] Apache Hadoop Ozone 0.5.0-beta RC1

[Arpit Agarwal]

HDDS-3116 is now fixed in the ozone-0.5.0 branch.

Folks - any more potential blockers before Dinesh spins RC2? I don’t see 
anything in Jira at the moment:

https://issues.apache.org/jira/issues/?jql=project%20in%20(%22HDDS%22)%20and%20%22Target%20Version%2Fs%22%20in%20(0.5.0)%20and%20resolution%20in%20(Unresolved)%20and%20priority%20in%20(Blocker)
 
<https://issues.apache.org/jira/issues/?jql=project%20in%20(%22HDDS%22)%20and%20%22Target%20Version/s%22%20in%20(0.5.0)%20and%20resolution%20in%20(Unresolved)%20and%20priority%20in%20(Blocker)>

Thanks,
Arpit


> On Mar 9, 2020, at 6:15 PM, Shashikant Banerjee 
>  wrote:
> 
> I think https://issues.apache.org/jira/browse/HDDS-3116 
> <https://issues.apache.org/jira/browse/HDDS-3116> is a blocker for
> the release. Because of this, datanodes fail to communicate with SCM and
> marked dead and don't seem to recover.
> This has been observed in multiple test setups.
> 
> Thanks
> Shashi
> 
> On Mon, Mar 9, 2020 at 9:20 PM Attila Doroszlai  <mailto:adorosz...@apache.org>>
> wrote:
> 
>> +1
>> 
>> * Verified GPG signature and SHA512 checksum
>> * Compiled sources
>> * Ran ozone smoke test against both binary and locally compiled versions
>> 
>> Thanks Dinesh for RC1.
>> 
>> -Attila
>> 
>> On Sun, Mar 8, 2020 at 2:34 AM Arpit Agarwal
>>  wrote:
>>> 
>>> +1 (binding)
>>> Verified mds, sha512
>>> Verified signatures
>>> Built from source
>>> Deployed to 3 node cluster
>>> Tried a few ozone shell and filesystem commands
>>> Ran freon load generator
>>> Thanks Dinesh for putting the RC1 together.
>>> 
>>> 
>>> 
>>>> On Mar 6, 2020, at 4:46 PM, Dinesh Chitlangia 
>> wrote:
>>>> 
>>>> Hi Folks,
>>>> 
>>>> We have put together RC1 for Apache Hadoop Ozone 0.5.0-beta.
>>>> 
>>>> The RC artifacts are at:
>>>> https://home.apache.org/~dineshc/ozone-0.5.0-rc1/
>>>> 
>>>> The public key used for signing the artifacts can be found at:
>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>>> 
>>>> The maven artifacts are staged at:
>>>> 
>> https://repository.apache.org/content/repositories/orgapachehadoop-1260
>>>> 
>>>> The RC tag in git is at:
>>>> https://github.com/apache/hadoop-ozone/tree/ozone-0.5.0-beta-RC1
>>>> 
>>>> This release contains 800+ fixes/improvements [1].
>>>> Thanks to everyone who put in the effort to make this happen.
>>>> 
>>>> *The vote will run for 7 days, ending on March 13th 2020 at 11:59 pm
>> PST.*
>>>> 
>>>> Note: This release is beta quality, it’s not recommended to use in
>>>> production but we believe that it’s stable enough to try out the
>> feature
>>>> set and collect feedback.
>>>> 
>>>> 
>>>> [1] https://s.apache.org/ozone-0.5.0-fixed-issues
>>>> 
>>>> Thanks,
>>>> Dinesh Chitlangia
>>> 
>> 
>> -
>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org 
>> <mailto:hdfs-dev-unsubscr...@hadoop.apache.org>
>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org 
>> <mailto:hdfs-dev-h...@hadoop.apache.org>

Re: [VOTE] Apache Hadoop Ozone 0.5.0-beta RC1

[Arpit Agarwal]

+1 (binding)
Verified mds, sha512
Verified signatures
Built from source
Deployed to 3 node cluster
Tried a few ozone shell and filesystem commands
Ran freon load generator 
Thanks Dinesh for putting the RC1 together.



> On Mar 6, 2020, at 4:46 PM, Dinesh Chitlangia  wrote:
> 
> Hi Folks,
> 
> We have put together RC1 for Apache Hadoop Ozone 0.5.0-beta.
> 
> The RC artifacts are at:
> https://home.apache.org/~dineshc/ozone-0.5.0-rc1/
> 
> The public key used for signing the artifacts can be found at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> 
> The maven artifacts are staged at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1260
> 
> The RC tag in git is at:
> https://github.com/apache/hadoop-ozone/tree/ozone-0.5.0-beta-RC1
> 
> This release contains 800+ fixes/improvements [1].
> Thanks to everyone who put in the effort to make this happen.
> 
> *The vote will run for 7 days, ending on March 13th 2020 at 11:59 pm PST.*
> 
> Note: This release is beta quality, it’s not recommended to use in
> production but we believe that it’s stable enough to try out the feature
> set and collect feedback.
> 
> 
> [1] https://s.apache.org/ozone-0.5.0-fixed-issues
> 
> Thanks,
> Dinesh Chitlangia

Re: [VOTE] Apache Hadoop Ozone 0.5.0-beta RC0

[Arpit Agarwal]

Hi Dinesh,

Thanks for spinning up this RC! Looks like we still had ~15 issues that were 
tagged as Blockers for 0.5.0 in jira.

I’ve moved out most of them, however the remaining 4 look like must fixes.

https://issues.apache.org/jira/issues/?jql=project%20%3D%20%22HDDS%22%20and%20%22Target%20Version%2Fs%22%20in%20(0.5.0)%20and%20resolution%20%3D%20Unresolved%20and%20priority%20%3D%20Blocker
 


Thanks,
Arpit


> On Feb 27, 2020, at 8:23 PM, Dinesh Chitlangia  wrote:
> 
> Hi Folks,
> 
> We have put together RC0 for Apache Hadoop Ozone 0.5.0-beta.
> 
> The RC artifacts are at:
> https://home.apache.org/~dineshc/ozone-0.5.0-rc0/
> 
> The public key used for signing the artifacts can be found at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> 
> The maven artifacts are staged at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1259
> 
> The RC tag in git is at:
> https://github.com/apache/hadoop-ozone/tree/ozone-0.5.0-beta-RC0
> 
> This release contains 800+ fixes/improvements [1].
> Thanks to everyone who put in the effort to make this happen.
> 
> *The vote will run for 7 days, ending on March 4th 2020 at 11:59 pm PST.*
> 
> Note: This release is beta quality, it’s not recommended to use in
> production but we believe that it’s stable enough to try out the feature
> set and collect feedback.
> 
> 
> [1] https://s.apache.org/ozone-0.5.0-fixed-issues
> 
> Thanks,
> Dinesh Chitlangia

Re: [DISCUSS] Feature branch for HDFS-14978 In-place Erasure Coding Conversion

[Arpit Agarwal]

+1


> On Jan 23, 2020, at 2:51 PM, Jitendra Pandey  wrote:
> 
> +1 for the feature branch. 
> 
> On Thu, Jan 23, 2020 at 1:34 PM Wei-Chiu Chuang 
>  wrote:
> Hi we are working on a feature to improve Erasure Coding, and I would like
> to seek your opinion on creating a feature branch for it. (HDFS-14978
>  >)
> 
> Reason for a feature branch
> (1) it turns out we need to update NameNode layout version
> (2) It's a medium size project and we want to get this feature merged in
> its entirety.
> 
> Aravindan Vijayan and I are planning to work on this feature.
> 
> Thoughts?

Re: [VOTE] create ozone-dev and ozone-issues mailing lists

[Arpit Agarwal]

Thanks for kicking this off Marton. Submitted INFRA requests to create the 
following. The lists should be live soon.

- ozone-dev@h.a.o
- ozone-issues@h.a.o
- ozone-commits@h.a.o



> On Oct 31, 2019, at 3:32 AM, Elek, Marton  wrote:
> 
> 
> Thanks for all the votes and feedback.
> 
> The vote is passed with no -1 and with many +1
> 
> The mailing lists will be created soon and the notification settings will be 
> updated.
> 
> Thank you for your patience.
> Marton
> 
> 
> On 10/27/19 9:25 AM, Elek, Marton wrote:
>> As discussed earlier in the thread of "Hadoop-Ozone repository mailing list 
>> configurations" [1] I suggested to solve the current misconfiguration 
>> problem with creating separated mailing lists (dev/issues) for Hadoop Ozone.
>> It would have some additional benefit: for example it would make easier to 
>> follow the Ozone development and future plans.
>> Here I am starting a new vote thread (open for at least 72 hours) to collect 
>> more feedback about this.
>> Please express your opinion / vote.
>> Thanks a lot,
>> Marton
>> [1] 
>> https://lists.apache.org/thread.html/dc66a30f48a744534e748c418bf7ab6275896166ca5ade11560ebaef@%3Chdfs-dev.hadoop.apache.org%3E
>>  -
>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] create ozone-dev and ozone-issues mailing lists

[Arpit Agarwal]

+1

> On Oct 27, 2019, at 1:25 AM, Elek, Marton  wrote:
> 
> 
> As discussed earlier in the thread of "Hadoop-Ozone repository mailing list 
> configurations" [1] I suggested to solve the current misconfiguration problem 
> with creating separated mailing lists (dev/issues) for Hadoop Ozone.
> 
> It would have some additional benefit: for example it would make easier to 
> follow the Ozone development and future plans.
> 
> Here I am starting a new vote thread (open for at least 72 hours) to collect 
> more feedback about this.
> 
> Please express your opinion / vote.
> 
> Thanks a lot,
> Marton
> 
> [1] 
> https://lists.apache.org/thread.html/dc66a30f48a744534e748c418bf7ab6275896166ca5ade11560ebaef@%3Chdfs-dev.hadoop.apache.org%3E
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] Separate Hadoop Core trunk and Hadoop Ozone trunk source tree

[Arpit Agarwal]

+1


> On Sep 17, 2019, at 2:49 AM, Elek, Marton  wrote:
> 
> 
> 
> TLDR; I propose to move Ozone related code out from Hadoop trunk and store it 
> in a separated *Hadoop* git repository apache/hadoop-ozone.git
> 
> 
> 
> 
> When Ozone was adopted as a new Hadoop subproject it was proposed[1] to be 
> part of the source tree but with separated release cadence, mainly because it 
> had the hadoop-trunk/SNAPSHOT as compile time dependency.
> 
> During the last Ozone releases this dependency is removed to provide more 
> stable releases. Instead of using the latest trunk/SNAPSHOT build from 
> Hadoop, Ozone uses the latest stable Hadoop (3.2.0 as of now).
> 
> As we have no more strict dependency between Hadoop trunk SNAPSHOT and Ozone 
> trunk I propose to separate the two code base from each other with creating a 
> new Hadoop git repository (apache/hadoop-ozone.git):
> 
> With moving Ozone to a separated git repository:
> 
> * It would be easier to contribute and understand the build (as of now we 
> always need `-f pom.ozone.xml` as a Maven parameter)
> * It would be possible to adjust build process without breaking Hadoop/Ozone 
> builds.
> * It would be possible to use different Readme/.asf.yaml/github template for 
> the Hadoop Ozone and core Hadoop. (For example the current github template 
> [2] has a link to the contribution guideline [3]. Ozone has an extended 
> version [4] from this guideline with additional information.)
> * Testing would be more safe as it won't be possible to change core Hadoop 
> and Hadoop Ozone in the same patch.
> * It would be easier to cut branches for Hadoop releases (based on the 
> original consensus, Ozone should be removed from all the release branches 
> after creating relase branches from trunk)
> 
> 
> What do you think?
> 
> Thanks,
> Marton
> 
> [1]: 
> https://lists.apache.org/thread.html/c85e5263dcc0ca1d13cbbe3bcfb53236784a39111b8c353f60582eb4@%3Chdfs-dev.hadoop.apache.org%3E
> [2]: 
> https://github.com/apache/hadoop/blob/trunk/.github/pull_request_template.md
> [3]: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
> [4]: 
> https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute+to+Ozone
> 
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Move Submarine source code, documentation, etc. to a separate Apache Git repo

[Arpit Agarwal]

+1

> On Aug 23, 2019, at 7:05 PM, Wangda Tan  wrote:
> 
> Hi devs,
> 
> This is a voting thread to move Submarine source code, documentation from
> Hadoop repo to a separate Apache Git repo. Which is based on discussions of
> https://lists.apache.org/thread.html/e49d60b2e0e021206e22bb2d430f4310019a8b29ee5020f3eea3bd95@%3Cyarn-dev.hadoop.apache.org%3E
> 
> Contributors who have permissions to push to Hadoop Git repository will
> have permissions to push to the new Submarine repository.
> 
> This voting thread will run for 7 days and will end at Aug 30th.
> 
> Please let me know if you have any questions.
> 
> Thanks,
> Wangda Tan


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: VOTE: Hadoop Ozone 0.4.0-alpha RC2

[Arpit Agarwal]

Thanks for building this RC Ajay.

+1 binding.

- verified signatures and checksums
- built from source 
- deployed to 3 node cluster, tried out basic operations
- ran smoke tests
- ran unit tests
- LICENSE/NOTICE files look ok

There is an extra file in the source root named JenkinsFile.



> On Apr 29, 2019, at 9:04 PM, Ajay Kumar  wrote:
> 
> Hi All,
> 
> 
> 
> We have created the third release candidate (RC2) for Apache Hadoop Ozone 
> 0.4.0-alpha.
> 
> 
> 
> This release contains security payload for Ozone. Below are some important 
> features in it:
> 
> 
> 
>  *   Hadoop Delegation Tokens and Block Tokens supported for Ozone.
>  *   Transparent Data Encryption (TDE) Support - Allows data blocks to be 
> encrypted-at-rest.
>  *   Kerberos support for Ozone.
>  *   Certificate Infrastructure for Ozone  - Tokens use PKI instead of shared 
> secrets.
>  *   Datanode to Datanode communication secured via mutual TLS.
>  *   Ability secure ozone cluster that works with Yarn, Hive, and Spark.
>  *   Skaffold support to deploy Ozone clusters on K8s.
>  *   Support S3 Authentication Mechanisms like - S3 v4 Authentication 
> protocol.
>  *   S3 Gateway supports Multipart upload.
>  *   S3A file system is tested and supported.
>  *   Support for Tracing and Profiling for all Ozone components.
>  *   Audit Support - including Audit Parser tools.
>  *   Apache Ranger Support in Ozone.
>  *   Extensive failure testing for Ozone.
> 
> The RC artifacts are available at 
> https://home.apache.org/~ajay/ozone-0.4.0-alpha-rc2/
> 
> 
> 
> The RC tag in git is ozone-0.4.0-alpha-RC2 (git hash 
> 4ea602c1ee7b5e1a5560c6cbd096de4b140f776b)
> 
> 
> 
> Please try 
> out,
>  vote, or just give us feedback.
> 
> 
> 
> The vote will run for 5 days, ending on May 4, 2019, 04:00 UTC.
> 
> 
> 
> Thank you very much,
> 
> Ajay


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: VOTE: Hadoop Ozone 0.4.0-alpha RC1

[Arpit Agarwal]

Thanks Ajay for putting together this RC.

Unfortunately HDDS-1425  looks 
like a blocker. We should make the docker experience smooth for anyone trying 
out 0.4.0.

I’ve just committed Marton’s patch for HDDS-1425 this morning. Let’s roll a new 
RC.



> On Apr 15, 2019, at 4:09 PM, Ajay Kumar  wrote:
> 
> Hi all,
> 
> We have created the second release candidate (RC1) for Apache Hadoop Ozone 
> 0.4.0-alpha.
> 
> This release contains security payload for Ozone. Below are some important 
> features in it:
> 
>  *   Hadoop Delegation Tokens and Block Tokens supported for Ozone.
>  *   Transparent Data Encryption (TDE) Support - Allows data blocks to be 
> encrypted-at-rest.
>  *   Kerberos support for Ozone.
>  *   Certificate Infrastructure for Ozone  - Tokens use PKI instead of shared 
> secrets.
>  *   Datanode to Datanode communication secured via mutual TLS.
>  *   Ability secure ozone cluster that works with Yarn, Hive, and Spark.
>  *   Skaffold support to deploy Ozone clusters on K8s.
>  *   Support S3 Authentication Mechanisms like - S3 v4 Authentication 
> protocol.
>  *   S3 Gateway supports Multipart upload.
>  *   S3A file system is tested and supported.
>  *   Support for Tracing and Profiling for all Ozone components.
>  *   Audit Support - including Audit Parser tools.
>  *   Apache Ranger Support in Ozone.
>  *   Extensive failure testing for Ozone.
> 
> The RC artifacts are available at 
> https://home.apache.org/~ajay/ozone-0.4.0-alpha-rc1
> 
> The RC tag in git is ozone-0.4.0-alpha-RC1 (git hash 
> d673e16d14bb9377f27c9017e2ffc1bcb03eebfb)
> 
> Please try 
> out,
>  vote, or just give us feedback.
> 
> The vote will run for 5 days, ending on April 20, 2019, 19:00 UTC.
> 
> Thank you very much,
> 
> Ajay
> 
> 

Re: [DISCUSS] Docker build process

[Arpit Agarwal]

Hi Eric,

> Dockerfile is most likely to change to apply the security fix.

I am not sure this is always. Marton’s point about revising docker images 
independent of Hadoop versions is valid. 


> When maven release is automated through Jenkins, this is a breeze
> of clicking a button.  Jenkins even increment the target version
> automatically with option to edit. 

I did not understand this suggestion. Could you please explain in simpler terms 
or share a link to the description?


> I will make adjustment accordingly unless 7 more people comes
> out and say otherwise.

What adjustment is this?

Thanks,
Arpit


> On Mar 19, 2019, at 10:19 AM, Eric Yang  wrote:
> 
> Hi Marton,
> 
> Thank you for your input.  I agree with most of what you said with a few 
> exceptions.  Security fix should result in a different version of the image 
> instead of replace of a certain version.  Dockerfile is most likely to change 
> to apply the security fix.  If it did not change, the source has instability 
> over time, and result in non-buildable code over time.  When maven release is 
> automated through Jenkins, this is a breeze of clicking a button.  Jenkins 
> even increment the target version automatically with option to edit.  It 
> makes release manager's job easier than Homer Simpson's job.
> 
> If versioning is done correctly, older branches can have the same docker 
> subproject, and Hadoop 2.7.8 can be released for older Hadoop branches.  We 
> don't generate timeline paradox to allow changing the history of Hadoop 
> 2.7.1.  That release has passed and let it stay that way.
> 
> There are mounting evidence that Hadoop community wants docker profile for 
> developer image.  Precommit build will not catch some build errors because 
> more codes are allowed to slip through using profile build process.  I will 
> make adjustment accordingly unless 7 more people comes out and say otherwise.
> 
> Regards,
> Eric
> 
> On 3/19/19, 1:18 AM, "Elek, Marton"  wrote:
> 
> 
> 
>Thank you Eric to describe the problem.
> 
>I have multiple small comments, trying to separate them.
> 
>I. separated vs in-build container image creation
> 
>> The disadvantages are:
>> 
>> 1.  Require developer to have access to docker.
>> 2.  Default build takes longer.
> 
> 
>These are not the only disadvantages (IMHO) as I wrote it in in the
>previous thread and the issue [1]
> 
>Using in-build container image creation doesn't enable:
> 
>1. to modify the image later (eg. apply security fixes to the container
>itself or apply improvements for the startup scripts)
>2. create images for older releases (eg. hadoop 2.7.1)
> 
>I think there are two kind of images:
> 
>a) images for released artifacts
>b) developer images
> 
>I would prefer to manage a) with separated branch repositories but b)
>with (optional!) in-build process.
> 
>II. Agree with Steve. I think it's better to make it optional as most of
>the time it's not required. I think it's better to support the default
>dev build with the default settings (=just enough to start)
> 
>III. Maven best practices
> 
>(https://dzone.com/articles/maven-profile-best-practices)
> 
>I think this is a good article. But this is not against profiles but
>creating multiple versions from the same artifact with the same name
>(eg. jdk8/jdk11). In Hadoop, profiles are used to introduce optional
>steps. I think it's fine as the maven lifecycle/phase model is very
>static (compare it with the tree based approach in Gradle).
> 
>Marton
> 
>[1]: https://issues.apache.org/jira/browse/HADOOP-16091
> 
>On 3/13/19 11:24 PM, Eric Yang wrote:
>> Hi Hadoop developers,
>> 
>> In the recent months, there were various discussions on creating docker 
>> build process for Hadoop.  There was convergence to make docker build 
>> process inline in the mailing list last month when Ozone team is planning 
>> new repository for Hadoop/ozone docker images.  New feature has started to 
>> add docker image build process inline in Hadoop build.
>> A few lessons learnt from making docker build inline in YARN-7129.  The 
>> build environment must have docker to have a successful docker build.  
>> BUILD.txt stated for easy build environment use Docker.  There is logic in 
>> place to ensure that absence of docker does not trigger docker build.  The 
>> inline process tries to be as non-disruptive as possible to existing 
>> development environment with one exception.  If docker’s presence is 
>> detected, but user does not have rights to run docker.  This will cause the 
>> build to fail.
>> 
>> Now, some developers are pushing back on inline docker build process because 
>> existing environment did not make docker build process mandatory.  However, 
>> there are benefits to use inline docker build process.  The listed benefits 
>> are:
>> 
>> 1.  Source code tag, maven repository artifacts and docker hub artifacts can 
>> all be pr

Re: [VOTE] Release Apache Hadoop 3.1.2 - RC1

[Arpit Agarwal]

+1 binding for updated source package.

  - Rechecked signatures and checksums
  - Source matches release git tag
  - Built from source


> On Feb 5, 2019, at 10:50 AM, Sunil G  wrote:
> 
> Thanks Billie for pointing out.
> I have updated source by removing patchprocess and extra line create
> release.
> 
> Also updated checksum as well.
> 
> @bil...@apache.org   @Wangda Tan 
> please help to verify this changed bit once.
> 
> Thanks
> Sunil
> 
> On Tue, Feb 5, 2019 at 5:23 AM Billie Rinaldi 
> wrote:
> 
>> Hey Sunil and Wangda, thanks for the RC. The source tarball has a
>> patchprocess directory with some yetus code in it. Also, the file
>> dev-support/bin/create-release file has the following line added:
>>  export GPG_AGENT_INFO="/home/sunilg/.gnupg/S.gpg-agent:$(pgrep
>> gpg-agent):1"
>> 
>> I think we are probably due for an overall review of LICENSE and NOTICE. I
>> saw some idiosyncrasies there but nothing that looked like a blocker.
>> 
>> On Mon, Jan 28, 2019 at 10:20 PM Sunil G  wrote:
>> 
>>> Hi Folks,
>>> 
>>> On behalf of Wangda, we have an RC1 for Apache Hadoop 3.1.2.
>>> 
>>> The artifacts are available here:
>>> http://home.apache.org/~sunilg/hadoop-3.1.2-RC1/
>>> 
>>> The RC tag in git is release-3.1.2-RC1:
>>> https://github.com/apache/hadoop/commits/release-3.1.2-RC1
>>> 
>>> The maven artifacts are available via repository.apache.org at
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1215
>>> 
>>> This vote will run 5 days from now.
>>> 
>>> 3.1.2 contains 325 [1] fixed JIRA issues since 3.1.1.
>>> 
>>> We have done testing with a pseudo cluster and distributed shell job.
>>> 
>>> My +1 to start.
>>> 
>>> Best,
>>> Wangda Tan and Sunil Govindan
>>> 
>>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.2)
>>> ORDER BY priority DESC
>>> 
>> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.1.2 - RC1

[Arpit Agarwal]

+1 (binding)

- Verified signatures
- Verified checksums
- Built from source
- Verified Maven artifacts on staging repo
- Deployed 3 node cluster
- Tried out HDFS commands, MapReduce jobs.

Confirmed the issues Billie pointed out. Not sure if you need to spin up a new 
RC or you can update the tarball - contents of the git tag look fine.


> On Jan 28, 2019, at 10:19 PM, Sunil G  wrote:
> 
> Hi Folks,
> 
> On behalf of Wangda, we have an RC1 for Apache Hadoop 3.1.2.
> 
> The artifacts are available here:
> http://home.apache.org/~sunilg/hadoop-3.1.2-RC1/
> 
> The RC tag in git is release-3.1.2-RC1:
> https://github.com/apache/hadoop/commits/release-3.1.2-RC1
> 
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1215
> 
> This vote will run 5 days from now.
> 
> 3.1.2 contains 325 [1] fixed JIRA issues since 3.1.1.
> 
> We have done testing with a pseudo cluster and distributed shell job.
> 
> My +1 to start.
> 
> Best,
> Wangda Tan and Sunil Govindan
> 
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.2)
> ORDER BY priority DESC


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: proposed new repository for hadoop/ozone docker images (+update on docker works)

[Arpit Agarwal]

I’ve requested a new repo hadoop-docker-ozone.git in gitbox.


> On Jan 22, 2019, at 4:59 AM, Elek, Marton  wrote:
> 
> 
> 
> TLDR;
> 
> I proposed to create a separated git repository for ozone docker images
> in HDDS-851 (hadoop-docker-ozone.git)
> 
> If there is no objections in the next 3 days I will ask an Apache Member
> to create the repository.
> 
> 
> 
> 
> LONG VERSION:
> 
> In HADOOP-14898 multiple docker containers and helper scripts are
> created for Hadoop.
> 
> The main goal was to:
> 
> 1.) help the development with easy-to-use docker images
> 2.) provide official hadoop images to make it easy to test new features
> 
> As of now we have:
> 
> - apache/hadoop-runner image (which contains the required dependency
> but no hadoop)
> - apache/hadoop:2 and apache/hadoop:3 images (to try out latest hadoop
> from 2/3 lines)
> 
> The base image to run hadoop (apache/hadoop-runner) is also heavily used
> for Ozone distribution/development.
> 
> The Ozone distribution contains docker-compose based cluster definitions
> to start various type of clusters and scripts to do smoketesting. (See
> HADOOP-16063 for more details).
> 
> Note: I personally believe that these definitions help a lot to start
> different type of clusters. For example it could be tricky to try out
> router based federation as it requires multiple HA clusters. But with a
> simple docker-compose definition [1] it could be started under 3
> minutes. (HADOOP-16063 is about creating these definitions for various
> hdfs/yarn use cases)
> 
> As of now we have dedicated branches in the hadoop git repository for
> the docker images (docker-hadoop-runner, docker-hadoop-2,
> docker-hadoop-3). It turns out that a separated repository would be more
> effective as the dockerhub can use only full branch names as tags.
> 
> We would like to provide ozone docker images to make the evaluation as
> easy as 'docker run -d apache/hadoop-ozone:0.3.0', therefore in HDDS-851
> we agreed to create a separated repository for the hadoop-ozone docker
> images.
> 
> If this approach works well we can also move out the existing
> docker-hadoop-2/docker-hadoop-3/docker-hadoop-runner branches from
> hadoop.git to an other separated hadoop-docker.git repository)
> 
> Please let me know if you have any comments,
> 
> Thanks,
> Marton
> 
> 1: see
> https://github.com/flokkr/runtime-compose/tree/master/hdfs/routerfeder
> as an example
> 
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop Ozone 0.3.0-alpha (RC1)

[Arpit Agarwal]

+1 binding.

  - Verified signatures
  - Verified checksums
  - Checked LICENSE/NOTICE files
  - Built from source
  - Deployed to three node cluster and ran smoke tests.

Thanks Marton for putting up the RC.


On 2018/11/14, 9:14 AM, "Elek, Marton"  wrote:

Hi all,

I've created the second release candidate (RC1) for Apache Hadoop Ozone
0.3.0-alpha including one more fix on top of the previous RC0 (HDDS-854)

This is the second release of Apache Hadoop Ozone. Notable changes since
the first release:

* A new S3 compatible rest server is added. Ozone can be used from any
S3 compatible tools (HDDS-434)
* Ozone Hadoop file system URL prefix is renamed from o3:// to o3fs://
(HDDS-651)
* Extensive testing and stability improvements of OzoneFs.
* Spark, YARN and Hive support and stability improvements.
* Improved Pipeline handling and recovery.
* Separated/dedicated classpath definitions for all the Ozone
components. (HDDS-447)

The RC artifacts are available from:
https://home.apache.org/~elek/ozone-0.3.0-alpha-rc1/

The RC tag in git is: ozone-0.3.0-alpha-RC1 (ebbf459e6a6)

Please try it out, vote, or just give us feedback.

The vote will run for 5 days, ending on November 19, 2018 18:00 UTC.


Thank you very much,
Marton


PS:

The easiest way to try it out is:

1. Download the binary artifact
2. Read the docs from ./docs/index.html
3. TLDR; cd compose/ozone && docker-compose up -d
4. open localhost:9874 or localhost:9876



The easiest way to try it out from the source:

1. mvn  install -DskipTests -Pdist -Dmaven.javadoc.skip=true -Phdds
-DskipShade -am -pl :hadoop-ozone-dist
2. cd hadoop-ozone/dist/target/ozone-0.3.0-alpha && docker-compose up -d



The easiest way to test basic functionality (with acceptance tests):

1. mvn  install -DskipTests -Pdist -Dmaven.javadoc.skip=true -Phdds
-DskipShade -am -pl :hadoop-ozone-dist
2. cd hadoop-ozone/dist/target/ozone-0.3.0-alpha/smoketest
3. ./test.sh

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop Ozone 0.2.1-alpha (RC0)

[Arpit Agarwal]

Thanks for putting together this release Marton.

+1 (binding)

  - Verified signatures and checksums
  - LICENSE and NOTICE files exist in source and binary tarballs
  - Built from source code
  - Deployed to three node cluster
  - Tried out Ozone shell commands via 'ozone sh'
  - Tried out OzoneFS commands using 'ozone fs'


Hit couple of issues, but they should not block Alpha1:
- SCM may not exit chill-mode if DataNodes started before it (startup order 
dependency?)
- DataNode logs are not going to the correct location.

-Arpit


On 2018/09/19, 2:49 PM, "Elek, Marton"  wrote:

Hi all,

After the recent discussion about the first Ozone release I've created 
the first release candidate (RC0) for Apache Hadoop Ozone 0.2.1-alpha.

This release is alpha quality: it’s not recommended to use in production 
but we believe that it’s stable enough to try it out the feature set and 
collect feedback.

The RC artifacts are available from: 
https://home.apache.org/~elek/ozone-0.2.1-alpha-rc0/

The RC tag in git is: ozone-0.2.1-alpha-RC0 (968082ffa5d)

Please try the release and vote; the vote will run for the usual 5 
working days, ending on September 26, 2018 10pm UTC time.

The easiest way to try it out is:

1. Download the binary artifact
2. Read the docs at ./docs/index.html
3. TLDR; cd compose/ozone && docker-compose up -d


Please try it out, vote, or just give us feedback.

Thank you very much,
Marton

ps: At next week, we will have a BoF session at ApacheCon North Europe, 
Montreal on Monday evening. Please join, if you are interested, or need 
support to try out the package or just have any feedback.


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: HADOOP-14163 proposal for new hadoop.apache.org

[Arpit Agarwal]

Requested a new git repo for the site: 
https://gitbox.apache.org/repos/asf/hadoop-site.git


On 9/4/18, 1:33 PM, "Mingliang Liu"  wrote:

It might be late but I'm +1 on the new site and transition proposal.

Thanks Marton.

On Fri, Aug 31, 2018 at 1:07 AM Elek, Marton  wrote:

> Bumping this thread at last time.
>
> I have the following proposal:
>
> 1. I will request a new git repository hadoop-site.git and import the
> new site to there (which has exactly the same content as the existing
> site).
>
> 2. I will ask infra to use the new repository as the source of
> hadoop.apache.org
>
> 3. I will sync manually all of the changes in the next two months back
> to the svn site from the git (release announcements, new committers)
>
> IN CASE OF ANY PROBLEM we can switch back to the svn without any problem.
>
> If no-one objects within three days, I'll assume lazy consensus and
> start with this plan. Please comment if you have objections.
>
> Again: it allows immediate fallback at any time as svn repo will be kept
> as is (+ I will keep it up-to-date in the next 2 months)
>
> Thanks,
> Marton
>
>
> On 06/21/2018 09:00 PM, Elek, Marton wrote:
> >
> > Thank you very much to bump up this thread.
> >
> >
> > About [2]: (Just for the clarification) the content of the proposed
> > website is exactly the same as the old one.
> >
> > About [1]. I believe that the "mvn site" is perfect for the
> > documentation but for website creation there are more simple and
> > powerful tools.
> >
> > Hugo has more simple compared to jekyll. Just one binary, without
> > dependencies, works everywhere (mac, linux, windows)
> >
> > Hugo has much more powerful compared to "mvn site". Easier to create/use
> > more modern layout/theme, and easier to handle the content (for example
> > new release announcements could be generated as part of the release
> > process)
> >
> > I think it's very low risk to try out a new approach for the site (and
> > easy to rollback in case of problems)
> >
> > Marton
> >
> > ps: I just updated the patch/preview site with the recent releases:
> >
> > ***
> > * http://hadoop.anzix.net *
> > ***
> >
> > On 06/21/2018 01:27 AM, Vinod Kumar Vavilapalli wrote:
> >> Got pinged about this offline.
> >>
> >> Thanks for keeping at it, Marton!
> >>
> >> I think there are two road-blocks here
> >>   (1) Is the mechanism using which the website is built good enough -
> >> mvn-site / hugo etc?
> >>   (2) Is the new website good enough?
> >>
> >> For (1), I just think we need more committer attention and get
> >> feedback rapidly and get it in.
> >>
> >> For (2), how about we do it in a different way in the interest of
> >> progress?
> >>   - We create a hadoop.apache.org/new-site/ where this new site goes.
> >>   - We then modify the existing web-site to say that there is a new
> >> site/experience that folks can click on a link and navigate to
> >>   - As this new website matures and gets feedback & fixes, we finally
> >> pull the plug at a later point of time when we think we are good to go.
> >>
> >> Thoughts?
> >>
> >> +Vinod
> >>
> >>> On Feb 16, 2018, at 3:10 AM, Elek, Marton  wrote:
> >>>
> >>> Hi,
> >>>
> >>> I would like to bump this thread up.
> >>>
> >>> TLDR; There is a proposed version of a new hadoop site which is
> >>> available from here: https://elek.github.io/hadoop-site-proposal/ and
> >>> https://issues.apache.org/jira/browse/HADOOP-14163
> >>>
> >>> Please let me know what you think about it.
> >>>
> >>>
> >>> Longer version:
> >>>
> >>> This thread started long time ago to use a more modern hadoop site:
> >>>
> >>> Goals were:
> >>>
> >>> 1. To make it easier to manage it (the release entries could be
> >>> created by a script as part of the release process)
> >>> 2. To use a better look-and-feel
> >>> 3. Move it out from svn to git
> >>>
> >>> I proposed to:
> >>>
> >>> 1. Move the existing site to git and generate it with hugo (which is
> >>> a single, standalone binary)
> >>> 2. Move both the rendered and source branches to git.
> >>> 3. (Create a jenkins job to generate the site automatically)
> >>>
> >>> NOTE: this is just about forrest based hadoop.apache.org, NOT about
> >>> the documentation which is generated by mvn-site (as before)
> >>>
> >>>
> >>> I got multiple valuable feedback and I improved the proposed site
> >>> according to the comments. Allen had some concerns about the used
> >>> technologies (hugo vs. mvn-site) and I answered all th

Re: HADOOP-14163 proposal for new hadoop.apache.org

[Arpit Agarwal]

+1

Thanks for initiating this Marton.


On 8/31/18, 1:07 AM, "Elek, Marton"  wrote:

Bumping this thread at last time.

I have the following proposal:

1. I will request a new git repository hadoop-site.git and import the 
new site to there (which has exactly the same content as the existing site).

2. I will ask infra to use the new repository as the source of 
hadoop.apache.org

3. I will sync manually all of the changes in the next two months back 
to the svn site from the git (release announcements, new committers)

IN CASE OF ANY PROBLEM we can switch back to the svn without any problem.

If no-one objects within three days, I'll assume lazy consensus and 
start with this plan. Please comment if you have objections.

Again: it allows immediate fallback at any time as svn repo will be kept 
as is (+ I will keep it up-to-date in the next 2 months)

Thanks,
Marton


On 06/21/2018 09:00 PM, Elek, Marton wrote:
> 
> Thank you very much to bump up this thread.
> 
> 
> About [2]: (Just for the clarification) the content of the proposed 
> website is exactly the same as the old one.
> 
> About [1]. I believe that the "mvn site" is perfect for the 
> documentation but for website creation there are more simple and 
> powerful tools.
> 
> Hugo has more simple compared to jekyll. Just one binary, without 
> dependencies, works everywhere (mac, linux, windows)
> 
> Hugo has much more powerful compared to "mvn site". Easier to create/use 
> more modern layout/theme, and easier to handle the content (for example 
> new release announcements could be generated as part of the release 
> process)
> 
> I think it's very low risk to try out a new approach for the site (and 
> easy to rollback in case of problems)
> 
> Marton
> 
> ps: I just updated the patch/preview site with the recent releases:
> 
> ***
> * http://hadoop.anzix.net *
> ***
> 
> On 06/21/2018 01:27 AM, Vinod Kumar Vavilapalli wrote:
>> Got pinged about this offline.
>>
>> Thanks for keeping at it, Marton!
>>
>> I think there are two road-blocks here
>>   (1) Is the mechanism using which the website is built good enough - 
>> mvn-site / hugo etc?
>>   (2) Is the new website good enough?
>>
>> For (1), I just think we need more committer attention and get 
>> feedback rapidly and get it in.
>>
>> For (2), how about we do it in a different way in the interest of 
>> progress?
>>   - We create a hadoop.apache.org/new-site/ where this new site goes.
>>   - We then modify the existing web-site to say that there is a new 
>> site/experience that folks can click on a link and navigate to
>>   - As this new website matures and gets feedback & fixes, we finally 
>> pull the plug at a later point of time when we think we are good to go.
>>
>> Thoughts?
>>
>> +Vinod
>>
>>> On Feb 16, 2018, at 3:10 AM, Elek, Marton  wrote:
>>>
>>> Hi,
>>>
>>> I would like to bump this thread up.
>>>
>>> TLDR; There is a proposed version of a new hadoop site which is 
>>> available from here: https://elek.github.io/hadoop-site-proposal/ and 
>>> https://issues.apache.org/jira/browse/HADOOP-14163
>>>
>>> Please let me know what you think about it.
>>>
>>>
>>> Longer version:
>>>
>>> This thread started long time ago to use a more modern hadoop site:
>>>
>>> Goals were:
>>>
>>> 1. To make it easier to manage it (the release entries could be 
>>> created by a script as part of the release process)
>>> 2. To use a better look-and-feel
>>> 3. Move it out from svn to git
>>>
>>> I proposed to:
>>>
>>> 1. Move the existing site to git and generate it with hugo (which is 
>>> a single, standalone binary)
>>> 2. Move both the rendered and source branches to git.
>>> 3. (Create a jenkins job to generate the site automatically)
>>>
>>> NOTE: this is just about forrest based hadoop.apache.org, NOT about 
>>> the documentation which is generated by mvn-site (as before)
>>>
>>>
>>> I got multiple valuable feedback and I improved the proposed site 
>>> according to the comments. Allen had some concerns about the used 
>>> technologies (hugo vs. mvn-site) and I answered all the questions why 
>>> I think mvn-site is the best for documentation and hugo is best for 
>>> generating site.
>>>
>>>
>>> I would like to finish this effort/jira: I would like to start a 
>>> discussion about using this proposed version and approach as a new 
>>> site of Apache Hadoop. Please let me know what you think.
>>>
>>>
>>> Thanks a lot,
>>> Marton
>>>
   

Re: [VOTE] reset/force push to clean up inadvertent merge commit pushed to trunk

[Arpit Agarwal]

Thanks Sunil!

@Subru and others who voted for force push, are you okay with cancelling this 
vote and declaring trunk open for commits?


On 7/6/18, 11:16 AM, "Sunil G"  wrote:

Thanks. These patches are now restored.

- Sunil


On Fri, Jul 6, 2018 at 11:14 AM Vinod Kumar Vavilapalli 
wrote:

> +1
>
> Thanks
> +Vinod
>
>
> On Jul 6, 2018, at 11:12 AM, Sunil G  wrote:
>
> I just checked.  YARN-7556 and YARN-7451 can be cherry-picked.
> I cherry-picked in my local and compiled. Things are good.
>
> I can push this now  which will restore trunk to its original.
> I can do this if there are no objection.
>
> - Sunil
    >
> On Fri, Jul 6, 2018 at 11:10 AM Arpit Agarwal 
> wrote:
>
> afaict YARN-8435 is still in trunk. YARN-7556 and YARN-7451 are not.
>
>
> From: Giovanni Matteo Fumarola 
> Date: Friday, July 6, 2018 at 10:59 AM
> To: Vinod Kumar Vavilapalli 
> Cc: Anu Engineer , Arpit Agarwal <
> aagar...@hortonworks.com>, "su...@apache.org" , "
> yarn-dev@hadoop.apache.org" , "
> hdfs-...@hadoop.apache.org" , "
> common-...@hadoop.apache.org" , "
> mapreduce-...@hadoop.apache.org" 
> Subject: Re: [VOTE] reset/force push to clean up inadvertent merge commit
> pushed to trunk
>
> Everything seems ok except the 3 commits: YARN-8435, YARN-7556, YARN-7451
> are not anymore in trunk due to the revert.
>
> Haibo/Robert if you can recommit your patches I will commit mine
> subsequently to preserve the original order.
>
> (My apology for the mess I did with the merge commit)
>
> On Fri, Jul 6, 2018 at 10:42 AM, Vinod Kumar Vavilapalli <
> vino...@apache.org<mailto:vino...@apache.org >> wrote:
> I will add that the branch also successfully compiles.
>
> Let's just move forward as is, unblock commits and just fix things if
> anything is broken.
>
> +Vinod
>
> On Jul 6, 2018, at 10:30 AM, Anu Engineer 
> <mailto:aengin...@hortonworks.com >> wrote:
>
>
> Hi All,
>
> [ Thanks to Arpit for working offline and verifying that branch is
>
> indeed good.]
>
>
> I want to summarize what I know of this issue and also solicit other
>
> points of view.
>
>
> We reverted the commit(c163d1797) from the branch, as soon as we noticed
>
> it. That is, we have made no other commits after the merge commit.
>
>
> We used the following command to revert
> git revert -c c163d1797ade0f47d35b4a44381b8ef1dfec5b60 -m 1
>
> Giovanni's branch had three commits + merge, The JIRAs he had were
>
> YARN-7451, YARN-7556, YARN-8435.
>
>
> The issue seems to be the revert of merge has some diffs. I am not a
>
> YARN developer, so the only problem is to look at the revert and see if
> there were any spurious edits in Giovanni's original commit + merge.
>
> If there are none, we don't need a reset/force push.  But if we find an
>
> issue I am more than willing to go the force commit route.
>
>
> The revert takes the trunk back to the point of the first commit from
>
> Giovanni which is YARN-8435. His branch was also rewriting the order of
> commits which we have lost due to the revert.
>
>
> Based on what I know so far, I am -1 on the force push.
>
> In other words, I am trying to understand why we need the force push. I
>
> have left a similar comment in JIRA (
> https://issues.apache.org/jira/browse/INFRA-16727) too.
>
>
>
> Thanks
> Anu
>
>
> On 7/6/18, 10:24 AM, "Arpit Agarwal" 
> aagar...@hortonworks.com>> wrote:
>
>
>   -1 for the force push. Nothing is broken in trunk. The history looks
>
> ugly for two commits and we can live with it.
>
>
>   The revert restored the branch to Giovanni's intent. i.e. only
>
> YARN-8435 is applied. Verified there is no delta between hashes 0d9804d 
and
> 39ad989 (HEAD).
>
>
>   39ad989 2018-07-05 aengineer@ o {apache/trunk} Revert "Merge branch
>
> 't...
>
>   c163d17 2018-07-05 gifuma@apa M─┐ Merge branch 'trunk' of
>
> https://git-...
>
>   99febe7 2018-07-05 rkanter@ap │ o YARN-7451. Add missing tests to
>
>

Re: [VOTE] reset/force push to clean up inadvertent merge commit pushed to trunk

[Arpit Agarwal]

+1 from me.

Also +1 to reopen trunk for all commits.


On 7/6/18, 11:12 AM, "Sunil G"  wrote:

I just checked.  YARN-7556 and YARN-7451 can be cherry-picked.
I cherry-picked in my local and compiled. Things are good.

I can push this now  which will restore trunk to its original.
I can do this if there are no objection.

- Sunil

On Fri, Jul 6, 2018 at 11:10 AM Arpit Agarwal 
wrote:

> afaict YARN-8435 is still in trunk. YARN-7556 and YARN-7451 are not.
>
>
> From: Giovanni Matteo Fumarola 
> Date: Friday, July 6, 2018 at 10:59 AM
> To: Vinod Kumar Vavilapalli 
    > Cc: Anu Engineer , Arpit Agarwal <
> aagar...@hortonworks.com>, "su...@apache.org" , "
> yarn-dev@hadoop.apache.org" , "
> hdfs-...@hadoop.apache.org" , "
> common-...@hadoop.apache.org" , "
> mapreduce-...@hadoop.apache.org" 
> Subject: Re: [VOTE] reset/force push to clean up inadvertent merge commit
> pushed to trunk
>
> Everything seems ok except the 3 commits: YARN-8435, YARN-7556, YARN-7451
> are not anymore in trunk due to the revert.
>
> Haibo/Robert if you can recommit your patches I will commit mine
> subsequently to preserve the original order.
>
> (My apology for the mess I did with the merge commit)
>
> On Fri, Jul 6, 2018 at 10:42 AM, Vinod Kumar Vavilapalli <
> vino...@apache.org<mailto:vino...@apache.org>> wrote:
> I will add that the branch also successfully compiles.
>
> Let's just move forward as is, unblock commits and just fix things if
> anything is broken.
>
> +Vinod
>
> > On Jul 6, 2018, at 10:30 AM, Anu Engineer  <mailto:aengin...@hortonworks.com>> wrote:
> >
> > Hi All,
> >
> > [ Thanks to Arpit for working offline and verifying that branch is
> indeed good.]
> >
> > I want to summarize what I know of this issue and also solicit other
> points of view.
> >
> > We reverted the commit(c163d1797) from the branch, as soon as we noticed
> it. That is, we have made no other commits after the merge commit.
> >
> > We used the following command to revert
> > git revert -c c163d1797ade0f47d35b4a44381b8ef1dfec5b60 -m 1
> >
> > Giovanni's branch had three commits + merge, The JIRAs he had were
> YARN-7451, YARN-7556, YARN-8435.
> >
> > The issue seems to be the revert of merge has some diffs. I am not a
> YARN developer, so the only problem is to look at the revert and see if
> there were any spurious edits in Giovanni's original commit + merge.
> > If there are none, we don't need a reset/force push.  But if we find an
> issue I am more than willing to go the force commit route.
> >
> > The revert takes the trunk back to the point of the first commit from
> Giovanni which is YARN-8435. His branch was also rewriting the order of
> commits which we have lost due to the revert.
> >
    > > Based on what I know so far, I am -1 on the force push.
> >
> > In other words, I am trying to understand why we need the force push. I
> have left a similar comment in JIRA (
> https://issues.apache.org/jira/browse/INFRA-16727) too.
> >
> >
> > Thanks
> > Anu
> >
> >
> > On 7/6/18, 10:24 AM, "Arpit Agarwal"  aagar...@hortonworks.com>> wrote:
> >
> >-1 for the force push. Nothing is broken in trunk. The history looks
> ugly for two commits and we can live with it.
> >
> >The revert restored the branch to Giovanni's intent. i.e. only
> YARN-8435 is applied. Verified there is no delta between hashes 0d9804d 
and
> 39ad989 (HEAD).
> >
> >39ad989 2018-07-05 aengineer@ o {apache/trunk} Revert "Merge branch
> 't...
> >c163d17 2018-07-05 gifuma@apa M─┐ Merge branch 'trunk' of
> https://git-...
> >99febe7 2018-07-05 rkanter@ap │ o YARN-7451. Add missing tests to
> veri...
> >1726247 2018-07-05 haibochen@ │ o YARN-7556. Fair scheduler
> configurat...
> >0d9804d 2018-07-05 gifuma@apa o │ YARN-8435. Fix NPE when the same
> cli...
> >71df8c2 2018-07-05 nanda@apac o─┘ HDDS-212. Introduce
> NodeStateManager...
> >
> >Regards,
> >Arpit
> >
> >
> >On 7/5/18, 2:37 PM, "Subru Krishnan"  su..

Re: [VOTE] reset/force push to clean up inadvertent merge commit pushed to trunk

[Arpit Agarwal]

afaict YARN-8435 is still in trunk. YARN-7556 and YARN-7451 are not.


From: Giovanni Matteo Fumarola 
Date: Friday, July 6, 2018 at 10:59 AM
To: Vinod Kumar Vavilapalli 
Cc: Anu Engineer , Arpit Agarwal 
, "su...@apache.org" , 
"yarn-dev@hadoop.apache.org" , 
"hdfs-...@hadoop.apache.org" , 
"common-...@hadoop.apache.org" , 
"mapreduce-...@hadoop.apache.org" 
Subject: Re: [VOTE] reset/force push to clean up inadvertent merge commit 
pushed to trunk

Everything seems ok except the 3 commits: YARN-8435, YARN-7556, YARN-7451 are 
not anymore in trunk due to the revert.

Haibo/Robert if you can recommit your patches I will commit mine subsequently 
to preserve the original order.

(My apology for the mess I did with the merge commit)

On Fri, Jul 6, 2018 at 10:42 AM, Vinod Kumar Vavilapalli 
mailto:vino...@apache.org>> wrote:
I will add that the branch also successfully compiles.

Let's just move forward as is, unblock commits and just fix things if anything 
is broken.

+Vinod

> On Jul 6, 2018, at 10:30 AM, Anu Engineer 
> mailto:aengin...@hortonworks.com>> wrote:
>
> Hi All,
>
> [ Thanks to Arpit for working offline and verifying that branch is indeed 
> good.]
>
> I want to summarize what I know of this issue and also solicit other points 
> of view.
>
> We reverted the commit(c163d1797) from the branch, as soon as we noticed it. 
> That is, we have made no other commits after the merge commit.
>
> We used the following command to revert
> git revert -c c163d1797ade0f47d35b4a44381b8ef1dfec5b60 -m 1
>
> Giovanni's branch had three commits + merge, The JIRAs he had were YARN-7451, 
> YARN-7556, YARN-8435.
>
> The issue seems to be the revert of merge has some diffs. I am not a YARN 
> developer, so the only problem is to look at the revert and see if there were 
> any spurious edits in Giovanni's original commit + merge.
> If there are none, we don't need a reset/force push.  But if we find an issue 
> I am more than willing to go the force commit route.
>
> The revert takes the trunk back to the point of the first commit from 
> Giovanni which is YARN-8435. His branch was also rewriting the order of 
> commits which we have lost due to the revert.
>
> Based on what I know so far, I am -1 on the force push.
>
> In other words, I am trying to understand why we need the force push. I have 
> left a similar comment in JIRA 
> (https://issues.apache.org/jira/browse/INFRA-16727) too.
>
>
> Thanks
> Anu
>
>
> On 7/6/18, 10:24 AM, "Arpit Agarwal" 
> mailto:aagar...@hortonworks.com>> wrote:
>
>-1 for the force push. Nothing is broken in trunk. The history looks ugly 
> for two commits and we can live with it.
>
>The revert restored the branch to Giovanni's intent. i.e. only YARN-8435 
> is applied. Verified there is no delta between hashes 0d9804d and 39ad989 
> (HEAD).
>
>39ad989 2018-07-05 aengineer@ o {apache/trunk} Revert "Merge branch 't...
>c163d17 2018-07-05 gifuma@apa M─┐ Merge branch 'trunk' of https://git-...
>99febe7 2018-07-05 rkanter@ap │ o YARN-7451. Add missing tests to veri...
>1726247 2018-07-05 haibochen@ │ o YARN-7556. Fair scheduler configurat...
>0d9804d 2018-07-05 gifuma@apa o │ YARN-8435. Fix NPE when the same cli...
>71df8c2 2018-07-05 nanda@apac o─┘ HDDS-212. Introduce NodeStateManager...
>
>Regards,
>Arpit
>
>
>On 7/5/18, 2:37 PM, "Subru Krishnan" 
> mailto:su...@apache.org>> wrote:
>
>Folks,
>
>There was a merge commit accidentally pushed to trunk, you can find the
>details in the mail thread [1].
>
>I have raised an INFRA ticket [2] to reset/force push to clean up 
> trunk.
>
>Can we have a quick vote for INFRA sign-off to proceed as this is 
> blocking
>all commits?
>
>Thanks,
>Subru
>
>[1]
>
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201807.mbox/%3CCAHqguubKBqwfUMwhtJuSD7X1Bgfro_P6FV%2BhhFhMMYRaxFsF9Q%40mail.gmail.com%3E
>[2] https://issues.apache.org/jira/browse/INFRA-16727
>
>
>
>-
>To unsubscribe, e-mail: 
> common-dev-unsubscr...@hadoop.apache.org<mailto:common-dev-unsubscr...@hadoop.apache.org>
>For additional commands, e-mail: 
> common-dev-h...@hadoop.apache.org<mailto:common-dev-h...@hadoop.apache.org>
>
>
>
> -
> To unsubscribe, e-mail: 
> common-dev-unsubscr...@hadoop.apache.org<mailto:common-dev-unsubscr...@hadoop.apache.org>
> For additional commands, e-mail: 
> common-dev-h...@hadoop.apache.org<mailto:common-dev-h...@hadoop.apache.org>

-
To unsubscribe, e-mail: 
yarn-dev-unsubscr...@hadoop.apache.org<mailto:yarn-dev-unsubscr...@hadoop.apache.org>
For additional commands, e-mail: 
yarn-dev-h...@hadoop.apache.org<mailto:yarn-dev-h...@hadoop.apache.org>

Re: [VOTE] reset/force push to clean up inadvertent merge commit pushed to trunk

[Arpit Agarwal]

-1 for the force push. Nothing is broken in trunk. The history looks ugly for 
two commits and we can live with it.

The revert restored the branch to Giovanni's intent. i.e. only YARN-8435 is 
applied. Verified there is no delta between hashes 0d9804d and 39ad989 (HEAD).

39ad989 2018-07-05 aengineer@ o {apache/trunk} Revert "Merge branch 't...
c163d17 2018-07-05 gifuma@apa M─┐ Merge branch 'trunk' of https://git-...
99febe7 2018-07-05 rkanter@ap │ o YARN-7451. Add missing tests to veri...
1726247 2018-07-05 haibochen@ │ o YARN-7556. Fair scheduler configurat...
0d9804d 2018-07-05 gifuma@apa o │ YARN-8435. Fix NPE when the same cli...
71df8c2 2018-07-05 nanda@apac o─┘ HDDS-212. Introduce NodeStateManager...

Regards,
Arpit


On 7/5/18, 2:37 PM, "Subru Krishnan"  wrote:

Folks,

There was a merge commit accidentally pushed to trunk, you can find the
details in the mail thread [1].

I have raised an INFRA ticket [2] to reset/force push to clean up trunk.

Can we have a quick vote for INFRA sign-off to proceed as this is blocking
all commits?

Thanks,
Subru

[1]

http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201807.mbox/%3CCAHqguubKBqwfUMwhtJuSD7X1Bgfro_P6FV%2BhhFhMMYRaxFsF9Q%40mail.gmail.com%3E
[2] https://issues.apache.org/jira/browse/INFRA-16727

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Arpit Agarwal]

Thanks Wangda, I see the shaded jars now.

Are the repo jars required to be the same as the binary release? They don’t 
match right now, probably they got rebuilt.

+1 (binding), modulo that remaining question.

* Verified signatures
* Verified checksums for source and binary artefacts
* Sanity checked jars on r.a.o. 
* Built from source
* Deployed to 3 node secure cluster with NameNode HA
* Verified HDFS web UIs
* Tried out HDFS shell commands
* Ran sample MapReduce jobs

Thanks!


--
From: Wangda Tan 
Date: Monday, April 2, 2018 at 9:25 PM
To: Arpit Agarwal 
Cc: Gera Shegalov , Sunil G , 
"yarn-dev@hadoop.apache.org" , Hdfs-dev 
, Hadoop Common , 
"mapreduce-...@hadoop.apache.org" , Vinod 
Kumar Vavilapalli 
Subject: Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

As pointed by Arpit, the previously deployed shared jars are incorrect. Just 
redeployed jars and staged. @Arpit, could you please check the updated Maven 
repo? https://repository.apache.org/content/repositories/orgapachehadoop-1092 

Since the jars inside binary tarballs are correct 
(http://people.apache.org/~wangda/hadoop-3.1.0-RC1/). I think we don't need 
roll another RC, just update Maven repo should be sufficient. 

Best,
Wangda


On Mon, Apr 2, 2018 at 2:39 PM, Wangda Tan <mailto:wheele...@gmail.com> wrote:
Hi Arpit, 

Thanks for pointing out this.

I just removed all .md5 files from artifacts. I found md5 checksums still exist 
in .mds files and I didn't remove them from .mds file because it is generated 
by create-release script and Apache guidance is "should not" instead of "must 
not". Please let me know if you think they need to be removed as well. 

- Wangda



On Mon, Apr 2, 2018 at 1:37 PM, Arpit Agarwal <mailto:aagar...@hortonworks.com> 
wrote:
Thanks for putting together this RC, Wangda. 

The guidance from Apache is to omit MD5s, specifically:
  > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).

https://www.apache.org/dev/release-distribution#sigs-and-sums

 


On Apr 2, 2018, at 7:03 AM, Wangda Tan <mailto:wheele...@gmail.com> wrote:

Hi Gera,

It's my bad, I thought only src/bin tarball is enough.

I just uploaded all other things under artifact/ to
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/

Please let me know if you have any other comments.

Thanks,
Wangda


On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov <mailto:ger...@gmail.com> wrote:


Thanks, Wangda!

There are many more artifacts in previous votes, e.g., see
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ .  Among others the
site tarball is missing.

On Sun, Apr 1, 2018 at 11:54 PM Sunil G <mailto:sun...@apache.org> wrote:


Thanks Wangda for initiating the release.

I tested this RC built from source file.


  - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
UI.
  - Below feature sanity is done
 - Application priority
 - Application timeout
 - Intra Queue preemption with priority based
 - DS based affinity tests to verify placement constraints.
  - Tested basic NodeLabel scenarios.
 - Added couple of labels to few of nodes and behavior is coming
 correct.
 - Verified old UI  and new YARN UI for labels.
 - Submitted apps to labelled cluster and it works fine.
 - Also performed few cli commands related to nodelabel.
  - Test basic HA cases and seems correct.
  - Tested new YARN UI . All pages are getting loaded correctly.


- Sunil

On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan <mailto:wheele...@gmail.com> wrote:


Hi folks,

Thanks to the many who helped with this release since Dec 2017 [1].
We've

created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:

http://people.apache.org/~wangda/hadoop-3.1.0-RC1

The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d

The maven artifacts are available via http://repository.apache.org at
https://repository.apache.org/content/repositories/
orgapachehadoop-1090/

This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.

3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services,
Support

rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.

For 3.1.0 RC0 vote discussion, please see [3].

We’d like to use this as a starting release for 3.1.x [1], depending on
how

it goes, get it stabilized and potentially use a 3.1.1 in several weeks
as

the stable release.

We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.

My +1 to start.

Best,
Wangda/Vinod

[1]

https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd10

Re: [VOTE] Release Apache Hadoop 3.0.1 (RC1)

[Arpit Agarwal]

Hi Lei,

It looks like the release artefacts have dummy shaded jars. E.g.

Repository Path:  
/org/apache/hadoop/hadoop-client-runtime/3.0.1/hadoop-client-runtime-3.0.1.jar
Uploaded by:  lei
Size: 44.47 KB
Uploaded Date:Fri Mar 16 2018 15:50:42 GMT-0700 (PDT)
Last Modified:Fri Mar 16 2018 15:50:42 GMT-0700 (PDT)

https://repository.apache.org/index.html#view-repositories;releases~browsestorage~/org/apache/hadoop/hadoop-client-runtime/3.0.1/hadoop-client-runtime-3.0.1.jar

Am I looking at this wrong or is this supposed to be the shaded jar which is 
~20MB?

Thanks,
Arpit



On 3/23/18, 10:18 AM, "Lei Xu"  wrote:

Hi, All

Thanks everyone for voting! The vote passes successfully with 6
binding +1s, 7 non-binding +1s and no -1s.

I will work on the staging and releases.

Best,


On Fri, Mar 23, 2018 at 5:10 AM, Kuhu Shukla  
wrote:
> +1 (non-binding)
>
> Built from source.
> Installed on a pseudo distributed cluster.
> Ran word count job and basic hdfs commands.
>
> Thank you for the effort on this release.
>
> Regards,
> Kuhu
>
> On Thu, Mar 22, 2018 at 5:25 PM, Elek, Marton  wrote:
>
>>
>> +1 (non binding)
>>
>> I did a full build from source code, created a docker container and did
>> various basic level tests with robotframework based automation and
>> docker-compose based pseudo clusters[1].
>>
>> Including:
>>
>> * Hdfs federation smoke test
>> * Basic ViewFS configuration
>> * Yarn example jobs
>> * Spark example jobs (with and without yarn)
>> * Simple hive table creation
>>
>> Marton
>>
>>
>> [1]: https://github.com/flokkr/runtime-compose
>>
>> On 03/18/2018 05:11 AM, Lei Xu wrote:
>>
>>> Hi, all
>>>
>>> I've created release candidate RC-1 for Apache Hadoop 3.0.1
>>>
>>> Apache Hadoop 3.0.1 will be the first bug fix release for Apache
>>> Hadoop 3.0 release. It includes 49 bug fixes and security fixes, which
>>> include 12
>>> blockers and 17 are critical.
>>>
>>> Please note:
>>> * HDFS-12990. Change default NameNode RPC port back to 8020. It makes
>>> incompatible changes to Hadoop 3.0.0.  After 3.0.1 releases, Apache
>>> Hadoop 3.0.0 will be deprecated due to this change.
>>>
>>> The release page is:
>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release
>>>
>>> New RC is available at: http://home.apache.org/~lei/hadoop-3.0.1-RC1/
>>>
>>> The git tag is release-3.0.1-RC1, and the latest commit is
>>> 496dc57cc2e4f4da117f7a8e3840aaeac0c1d2d0
>>>
>>> The maven artifacts are available at:
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1081/
>>>
>>> Please try the release and vote; the vote will run for the usual 5
>>> days, ending on 3/22/2017 6pm PST time.
>>>
>>> Thanks!
>>>
>>> -
>>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>>>
>>>
>> -
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>
>>



-- 
Lei (Eddy) Xu
Software Engineer, Cloudera

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Arpit Agarwal]

Thanks for putting together this RC, Wangda.

The guidance from Apache is to omit MD5s, specifically:
  > SHOULD NOT supply a MD5 checksum file (because MD5 is too broken).

https://www.apache.org/dev/release-distribution#sigs-and-sums



On Apr 2, 2018, at 7:03 AM, Wangda Tan 
mailto:wheele...@gmail.com>> wrote:

Hi Gera,

It's my bad, I thought only src/bin tarball is enough.

I just uploaded all other things under artifact/ to
http://people.apache.org/~wangda/hadoop-3.1.0-RC1/

Please let me know if you have any other comments.

Thanks,
Wangda


On Mon, Apr 2, 2018 at 12:50 AM, Gera Shegalov  wrote:

Thanks, Wangda!

There are many more artifacts in previous votes, e.g., see
http://home.apache.org/~junping_du/hadoop-2.8.3-RC0/ .  Among others the
site tarball is missing.

On Sun, Apr 1, 2018 at 11:54 PM Sunil G  wrote:

Thanks Wangda for initiating the release.

I tested this RC built from source file.


  - Tested MR apps (sleep, wc) and verified both new YARN UI and old RM
UI.
  - Below feature sanity is done
 - Application priority
 - Application timeout
 - Intra Queue preemption with priority based
 - DS based affinity tests to verify placement constraints.
  - Tested basic NodeLabel scenarios.
 - Added couple of labels to few of nodes and behavior is coming
 correct.
 - Verified old UI  and new YARN UI for labels.
 - Submitted apps to labelled cluster and it works fine.
 - Also performed few cli commands related to nodelabel.
  - Test basic HA cases and seems correct.
  - Tested new YARN UI . All pages are getting loaded correctly.


- Sunil

On Fri, Mar 30, 2018 at 9:45 AM Wangda Tan  wrote:

Hi folks,

Thanks to the many who helped with this release since Dec 2017 [1].
We've
created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:

http://people.apache.org/~wangda/hadoop-3.1.0-RC1

The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d

The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/
orgapachehadoop-1090/
This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.

3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
include the first class GPU/FPGA support on YARN, Native services,
Support
rich placement constraints in YARN, S3-related enhancements, allow HDFS
block replicas to be provided by an external storage system, etc.

For 3.1.0 RC0 vote discussion, please see [3].

We’d like to use this as a starting release for 3.1.x [1], depending on
how
it goes, get it stabilized and potentially use a 3.1.1 in several weeks
as
the stable release.

We have done testing with a pseudo cluster:
- Ran distributed job.
- GPU scheduling/isolation.
- Placement constraints (intra-application anti-affinity) by using
distributed shell.

My +1 to start.

Best,
Wangda/Vinod

[1]

https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
[2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER
BY
fixVersion ASC
[3]

https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E

Re: About reset branch-3.1 to trunk before release.

[Arpit Agarwal]

Thanks Wangda.


On 3/19/18, 11:38 AM, "Wangda Tan"  wrote:

Done JIRA fix version update:

Moved all JIRAs with fixVersion = 3.2.0 to 3.1.0 except following few fixes
(which committed after 49c747ab187d0650143205ba57ca19607ec4c6bd)

YARN-8002. Support NOT_SELF and ALL namespace types for allocation tag.
(Weiwe
i Yang via wangda)

HADOOP-15262. AliyunOSS: move files under a directory in parallel when
rename a directory. Contributed by Jinhu Wu.

MAPREDUCE-7066. TestQueue fails on Java9

YARN-8028. Support authorizeUserAccessToQueue in RMWebServices.
Contributed by Wangda Tan.

YARN-8040. [UI2] New YARN UI webapp does not respect current pathname
for REST api. Contributed by Sunil G.

Thanks,
Wangda

On Mon, Mar 19, 2018 at 11:12 AM, Wangda Tan  wrote:

> Thanks Akira for the additional vote,
>
> With help from Apache Infra Team (Daniel Takamori), we just reset
> branch-3.1 to trunk (SHA: 49c747ab187d0650143205ba57ca19607ec4c6bd). Will
> update JIRA fix version shortly.
>
> - Wangda
>
> On Sun, Mar 18, 2018 at 6:10 PM, Akira Ajisaka  > wrote:
>
>> +1 for resetting branch-3.1.
>>
>> Thanks,
>> Akira
>>
>>
>> On 2018/03/18 12:51, Wangda Tan wrote:
>>
>>> Thanks for sharing your thoughts.
>>>
>>> We have done build and single node cluster deploy / test for the latest
>>> trunk code (commit: 49c747ab187d0650143205ba57ca19607ec4c6bd). Since
>>> there
>>> are no objections, so I will go ahead to do the branch replace.
>>>
>>> Since we don't have force push permission to release branches. I just
>>> filed
>>> https://issues.apache.org/jira/browse/INFRA-16204 to get help from
>>> Apache
>>> infra team.
>>>
>>> Please hold any commits to branch-3.1, will keep this email thread
>>> posted.
>>>
>>> Best,
>>> Wangda
>>>
>>> On Wed, Mar 14, 2018 at 3:14 PM, Vinod Kumar Vavilapalli <
>>> vino...@apache.org
>>>
 wrote:

>>>
>>> I see one new feature: https://issues.apache.org/jira/browse/YARN-7626:
 Allow regular expression matching in container-executor.cfg for devices
 and
 named docker volumes mount.

 There are 21 sub-tasks. There are three feature-type JIRAs in those -
 https://issues.apache.org/jira/browse/YARN-7972,
 https://issues.apache.org/jira/browse/YARN-7891 and
 https://issues.apache.org/jira/browse/YARN-5015. These should be okay -
 not major disrupting features.

 Everything else is either a bug-fix or an improvement so we should be
 good.

  From the list, it doesn't look like resetting will destabilize 3.1, +1
 for
 doing this.

 Thanks
 +Vinod

 On Mar 14, 2018, at 1:54 PM, Wangda Tan  wrote:
>
> Hi mapreduce/yarn/common/hdfs-devs,
>
> As of now, we have all blockers done for 3.1.0 release [1]. The 
release
>
 is running behind schedule due to a few security-related issues.
 Because of
 this and since branch-3.1 is cut 5 weeks before on Feb 8, trunk 3.2 is
 already diverging. There're 64 commits in trunk but not in branch-3.1.
 [2]

>
> I took a quick scan of them, most of them are good fixes which we
> should
>
 bring to 3.1.0 as well. And this can also reduce differences between
 3.2.0
 and 3.1.0 release for less maintenance burden in the future.

>
> Unless anyone objects, we will reset branch-3.1 to trunk in 1-2 days
> and
>
 cut RC after that.

>
> Thoughts?
>
> - Wangda
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in
> (Blocker,
>
 Critical) AND resolution = Unresolved AND "Target Version/s" = 3.1.0
 ORDER
 BY priority DESC

> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> (3.2.0)
>
 AND fixVersion not in (3.1.0)



>>>
>



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Apache Hadoop 3.0.1 Release plan

[Arpit Agarwal]

Hi Aaron/Lei,

Do you plan to roll an RC with an uncommitted fix? That isn't the right 
approach.

This issue has good visibility and enough discussion. If there is a binding 
veto in effect then the change must be abandoned. Else you should be able to 
proceed with committing. However, 3.0.0 must be called out as an abandoned 
release if we commit it.

Regards,
Arpit


On 2/1/18, 3:01 PM, "Lei Xu"  wrote:

Sounds good to me, ATM.

On Thu, Feb 1, 2018 at 2:34 PM, Aaron T. Myers  wrote:
> Hey Anu,
>
> My feeling on HDFS-12990 is that we've discussed it quite a bit already 
and
> it doesn't seem at this point like either side is going to budge. I'm
> certainly happy to have a phone call about it, but I don't expect that 
we'd
> make much progress.
>
> My suggestion is that we simply include the patch posted to HDFS-12990 in
> the 3.0.1 RC and call this issue out clearly in the subsequent VOTE thread
> for the 3.0.1 release. Eddy, are you up for that?
>
> Best,
> Aaron
>
> On Thu, Feb 1, 2018 at 1:13 PM, Lei Xu  wrote:
>>
>> +Xiao
>>
>> My understanding is that we will have this for 3.0.1.   Xiao, could
>> you give your inputs here?
>>
>> On Thu, Feb 1, 2018 at 11:55 AM, Anu Engineer 
>> wrote:
>> > Hi Eddy,
>> >
>> > Thanks for driving this release. Just a quick question, do we have time
>> > to close this issue?
>> > https://issues.apache.org/jira/browse/HDFS-12990
>> >
>> > or are we abandoning it? I believe that this is the last window for us
>> > to fix this issue.
>> >
>> > Should we have a call and get this resolved one way or another?
>> >
>> > Thanks
>> > Anu
>> >
>> > On 2/1/18, 10:51 AM, "Lei Xu"  wrote:
>> >
>> > Hi, All
>> >
>> > I just cut branch-3.0.1 from branch-3.0.  Please make sure all
>> > patches
>> > targeted to 3.0.1 being checked in both branch-3.0 and 
branch-3.0.1.
>> >
>> > Thanks!
>> > Eddy
>> >
>> > On Tue, Jan 9, 2018 at 11:17 AM, Lei Xu  wrote:
>> > > Hi, All
>> > >
>> > > We have released Apache Hadoop 3.0.0 in December [1]. To further
>> > > improve the quality of release, we plan to cut branch-3.0.1 
branch
>> > > tomorrow for the preparation of Apache Hadoop 3.0.1 release. The
>> > focus
>> > > of 3.0.1 will be fixing blockers (3), critical bugs (1) and bug
>> > fixes
>> > > [2].  No new features and improvement should be included.
>> > >
>> > > We plan to cut branch-3.0.1 tomorrow (Jan 10th) and vote for RC 
on
>> > Feb
>> > > 1st, targeting for Feb 9th release.
>> > >
>> > > Please feel free to share your insights.
>> > >
>> > > [1]
>> > https://www.mail-archive.com/general@hadoop.apache.org/msg07757.html
>> > > [2] https://issues.apache.org/jira/issues/?filter=12342842
>> > >
>> > > Best,
>> > > --
>> > > Lei (Eddy) Xu
>> > > Software Engineer, Cloudera
>> >
>> >
>> >
>> > --
>> > Lei (Eddy) Xu
>> > Software Engineer, Cloudera
>> >
>> >
>> > -
>> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>> >
>> >
>> >
>>
>>
>>
>> --
>> Lei (Eddy) Xu
>> Software Engineer, Cloudera
>>
>> -
>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>>
>



-- 
Lei (Eddy) Xu
Software Engineer, Cloudera

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [ANNOUNCE] Apache Hadoop 3.0.0 GA is released

[Arpit Agarwal]

That makes sense for Beta users but most of our users will be upgrading from a 
previous GA release and the changelog will mislead them. The webpage does not 
mention this is a delta from the beta release.


From: Andrew Wang 
Date: Friday, December 15, 2017 at 10:36 AM
To: Arpit Agarwal 
Cc: general , "common-...@hadoop.apache.org" 
, "yarn-dev@hadoop.apache.org" 
, "mapreduce-...@hadoop.apache.org" 
, "hdfs-...@hadoop.apache.org" 

Subject: Re: [ANNOUNCE] Apache Hadoop 3.0.0 GA is released

Hi Arpit,

If you look at the release announcements, it's made clear that the changelog 
for 3.0.0 is diffed based on beta1. This is important since users need to know 
what's different from the previous 3.0.0-* releases if they're upgrading.

I agree there's additional value to making combined release notes, but it'd be 
something additive rather than replacing what's there.

Best,
Andrew

On Fri, Dec 15, 2017 at 8:27 AM, Arpit Agarwal 
mailto:aagar...@hortonworks.com>> wrote:

Hi Andrew,

Thank you for all the hard work on this release. I was out the last few days 
and didn’t get a chance to evaluate RC1 earlier.

The changelog looks incorrect. E.g. This gives an impression that there are 
just 5 incompatible changes in 3.0.0.
http://hadoop.apache.org/docs/r3.0.0/hadoop-project-dist/hadoop-common/release/3.0.0/CHANGES.3.0.0.html

I assume you only counted 3.0.0 changes in this log excluding alphas/betas. 
However, users shouldn’t have to manually compile incompatibilities by summing 
up a/b release notes. Can we fix the changelog after the fact?




On 12/14/17, 10:45 AM, "Andrew Wang" 
mailto:andrew.w...@cloudera.com>> wrote:

Hi all,

I'm pleased to announce that Apache Hadoop 3.0.0 is generally available
(GA).

3.0.0 GA consists of 302 bug fixes, improvements, and other enhancements
since 3.0.0-beta1. This release marks a point of quality and stability for
the 3.0.0 release line, and users of earlier 3.0.0-alpha and -beta releases
are encouraged to upgrade.

Looking back, 3.0.0 GA is the culmination of over a year of work on the
3.0.0 line, starting with 3.0.0-alpha1 which was released in September
2016. Altogether, 3.0.0 incorporates 6,242 changes since 2.7.0.

Users are encouraged to read the overview of major changes
<http://hadoop.apache.org/docs/r3.0.0/index.html> in 3.0.0. The GA release
notes

<http://hadoop.apache.org/docs/r3.0.0/hadoop-project-dist/hadoop-common/release/3.0.0/RELEASENOTES.3.0.0.html>
 and changelog

<http://hadoop.apache.org/docs/r3.0.0/hadoop-project-dist/hadoop-common/release/3.0.0/CHANGES.3.0.0.html>
detail
the changes since 3.0.0-beta1.

The ASF press release provides additional color and highlights some of the
major features:


https://globenewswire.com/news-release/2017/12/14/1261879/0/en/The-Apache-Software-Foundation-Announces-Apache-Hadoop-v3-0-0-General-Availability.html

Let me end by thanking the many, many contributors who helped with this
release line. We've only had three major releases in Hadoop's 10 year
history, and this is our biggest major release ever. It's an incredible
accomplishment for our community, and I'm proud to have worked with all of
you.

Best,
Andrew

Re: [ANNOUNCE] Apache Hadoop 3.0.0 GA is released

[Arpit Agarwal]

Hi Andrew,

Thank you for all the hard work on this release. I was out the last few days 
and didn’t get a chance to evaluate RC1 earlier.

The changelog looks incorrect. E.g. This gives an impression that there are 
just 5 incompatible changes in 3.0.0.
http://hadoop.apache.org/docs/r3.0.0/hadoop-project-dist/hadoop-common/release/3.0.0/CHANGES.3.0.0.html

I assume you only counted 3.0.0 changes in this log excluding alphas/betas. 
However, users shouldn’t have to manually compile incompatibilities by summing 
up a/b release notes. Can we fix the changelog after the fact?




On 12/14/17, 10:45 AM, "Andrew Wang"  wrote:

Hi all,

I'm pleased to announce that Apache Hadoop 3.0.0 is generally available
(GA).

3.0.0 GA consists of 302 bug fixes, improvements, and other enhancements
since 3.0.0-beta1. This release marks a point of quality and stability for
the 3.0.0 release line, and users of earlier 3.0.0-alpha and -beta releases
are encouraged to upgrade.

Looking back, 3.0.0 GA is the culmination of over a year of work on the
3.0.0 line, starting with 3.0.0-alpha1 which was released in September
2016. Altogether, 3.0.0 incorporates 6,242 changes since 2.7.0.

Users are encouraged to read the overview of major changes
 in 3.0.0. The GA release
notes


 and changelog


detail
the changes since 3.0.0-beta1.

The ASF press release provides additional color and highlights some of the
major features:


https://globenewswire.com/news-release/2017/12/14/1261879/0/en/The-Apache-Software-Foundation-Announces-Apache-Hadoop-v3-0-0-General-Availability.html

Let me end by thanking the many, many contributors who helped with this
release line. We've only had three major releases in Hadoop's 10 year
history, and this is our biggest major release ever. It's an incredible
accomplishment for our community, and I'm proud to have worked with all of
you.

Best,
Andrew

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Arpit Agarwal]

Thanks for that proposal Andrew, and for not wrapping up the vote yesterday.

> In terms of downstream testing, we've done extensive
> integration testing with downstreams via the alphas
> and betas, and we have continuous integration running
> at Cloudera against branch-3.0.

Could you please share what kind of downstream testing you have performed and 
with which downstream projects?

Regards,
Arpit


On 11/17/17, 3:27 PM, "Andrew Wang"  wrote:

Hi Arpit,

I agree the timing is not great here, but extending it to meaningfully
avoid the holidays would mean extending it an extra week (e.g. to the
29th). We've been coordinating with ASF PR for that Tuesday, so I'd really,
really like to get the RC out before then.

In terms of downstream testing, we've done extensive integration testing
with downstreams via the alphas and betas, and we have continuous
integration running at Cloudera against branch-3.0. Because of this, I have
more confidence in our integration for 3.0.0 than most Hadoop releases.

Is it meaningful to extend to say, the 21st, which provides for a full week
of voting?

Best,
Andrew

    On Fri, Nov 17, 2017 at 1:27 PM, Arpit Agarwal 
wrote:

> Hi Andrew,
>
> Thank you for your hard work in getting us to this step. This is our first
> major GA release in many years.
>
> I feel a 5-day vote window ending over the weekend before thanksgiving may
> not provide sufficient time to evaluate this RC especially for downstream
> components.
>
> Would you please consider extending the voting deadline until a few days
> after the thanksgiving holiday? It would be a courtesy to our broader
> community and I see no harm in giving everyone a few days to evaluate it
> more thoroughly.
>
> On a lighter note, your deadline is also 4 minutes short of the required 5
> days. :)
>
> Regards,
> Arpit
>
>
>
> On 11/14/17, 1:34 PM, "Andrew Wang"  wrote:
>
> Hi folks,
>
> Thanks as always to the many, many contributors who helped with this
> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> available here:
>
> http://people.apache.org/~wang/3.0.0-RC0/
>
> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>
> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> additions include the merge of YARN resource types, API-based
> configuration
> of the CapacityScheduler, and HDFS router-based federation.
>
> I've done my traditional testing with a pseudo cluster and a Pi job.
> My +1
> to start.
>
> Best,
> Andrew
>
>
>

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Arpit Agarwal]

Hi Andrew,

Thank you for your hard work in getting us to this step. This is our first 
major GA release in many years.

I feel a 5-day vote window ending over the weekend before thanksgiving may not 
provide sufficient time to evaluate this RC especially for downstream 
components.

Would you please consider extending the voting deadline until a few days after 
the thanksgiving holiday? It would be a courtesy to our broader community and I 
see no harm in giving everyone a few days to evaluate it more thoroughly.

On a lighter note, your deadline is also 4 minutes short of the required 5 
days. :)

Regards,
Arpit



On 11/14/17, 1:34 PM, "Andrew Wang"  wrote:

Hi folks,

Thanks as always to the many, many contributors who helped with this
release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
available here:

http://people.apache.org/~wang/3.0.0-RC0/

This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.

3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
additions include the merge of YARN resource types, API-based configuration
of the CapacityScheduler, and HDFS router-based federation.

I've done my traditional testing with a pseudo cluster and a Pi job. My +1
to start.

Best,
Andrew

Re: [VOTE] Release cadence and EOL

[Arpit Agarwal]

The ASF release policy says releases may not be vetoed [1] so the EOL policy 
sounds unenforceable. Not sure a release cadence is enforceable either since 
Release Managers are volunteers.

1. https://www.apache.org/dev/release.html#approving-a-release



On 1/18/17, 7:06 PM, "Junping Du"  wrote:

+1 on Sangjin's proposal - 
"A minor release line is end-of-lifed 2 years after it is released or there
are 2 newer minor releases, whichever is sooner. The community reserves the
right to extend or shorten the life of a release line if there is a good
reason to do so."

I also noticed Karthik bring up some new proposals - some of them looks 
interesting to me and I have some ideas as well. Karthik, can you bring it out 
in a separated discussion threads so that we can discuss from there?

About Chris Trezzo's question about definition of EOL of hadoop release, I 
think potentially changes could be: 
1. For users of Apache hadoop, they would expect to upgrade to a new 
minor/major releases after EOL of their current release because there is no 
guarantee of new maintenance release.

2. For release effort, apache law claim that committer can volunteer RM for 
any release. With this release EOL proposal passes and written into hadoop 
bylaw, anyone want to call for a release which is EOL then she/he have to 
provide a good reason to community and get voted before to start release 
effort. We don't want to waste community time/resource to verify/vote a narrow 
interested release.

3. About committer's responsibility, I think the bottom line is committer 
should commit patch contributor's target release and her/his own interest 
release which I conservatively agree with Allen's point that this vote doesn't 
change anything. But if a committer want to take care more interest from the 
whole community like most committers are doing today, he/she should understand 
which branches can benefit more people and could skip some EOL release branches 
for backport effort.

About major release EOL, this could be more complicated and I think we 
should discuss separately.

Thanks,

Junping

From: Allen Wittenauer 
Sent: Wednesday, January 18, 2017 3:30 PM
To: Chris Trezzo
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: [VOTE] Release cadence and EOL

> On Jan 18, 2017, at 11:21 AM, Chris Trezzo  wrote:
>
> Thanks Sangjin for pushing this forward! I have a few questions:

These are great questions, because I know I'm not seeing a whole 
lot of substance in this vote.  The way to EOL software in the open source 
universe is with new releases and aging it out.  If someone wants to be a RE 
for a new branch-1 release, more power to them.  As volunteers to the ASF, 
we're not on the hook to provide much actual support.  This feels more like a 
vendor play than a community one.  But if the PMC want to vote on it, whatever. 
 It won't be first bylaw that doesn't really mean much.

> 1. What is the definition of end-of-life for a release in the hadoop
> project? My current understanding is as follows: When a release line
> reaches end-of-life, there are no more planned releases for that line.
> Committers are no longer responsible for back-porting bug fixes to the 
line
> (including fixed security vulnerabilities) and it is essentially
> unmaintained.

Just a point of clarification.  There is no policy that says that 
committers must back port.  It's up to the individual committers to push a 
change onto any particular branch. Therefore, this vote doesn't really change 
anything in terms of committer responsibilities here.

> 2. How do major releases affect the end-of-life proposal? For example, how
> does a new minor release in the next major release affect the end-of-life
> of minor releases in a previous major release? Is it possible to have a
> maintained 2.x release if there is a 3.3 release?

I'm looking forward to seeing this answer too, given that 2.7.0 is 
probably past the 2 year mark, 2.8.0 has seemingly been in a holding pattern 
for over a year, and the next 3.0.0 alpha should be RSN

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org


-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.2 RC2

[Arpit Agarwal]

+1 (binding)

- verified signatures
- Installed pseudo-cluster from binary distribution and ran example MapReduce 
jobs
- Built from sources and reran tests on pseudo-cluster




On 1/14/16, 8:57 PM, "Vinod Kumar Vavilapalli"  wrote:

>Hi all,
>
>I've created an updated release candidate RC2 for Apache Hadoop 2.7.2.
>
>As discussed before, this is the next maintenance release to follow up 2.7.1.
>
>The RC is available for validation at: 
>http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/
>
>The RC tag in git is: release-2.7.2-RC2
>
>The maven artifacts are available via repository.apache.org 
> at 
>https://repository.apache.org/content/repositories/orgapachehadoop-1027 
>
>
>The release-notes are inside the tar-balls at location 
>hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
>this at http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/releasenotes.html 
> for 
>your quick perusal.
>
>As you may have noted,
> - I terminated the RC1 related voting thread after finding out that we didn’t 
> have a bunch of patches that are already in the released 2.6.3 version. After 
> a brief discussion, we decided to keep the parallel 2.6.x and 2.7.x releases 
> incremental, see [4] for this discussion.
> - The RC0 related voting thread got halted due to some critical issues. It 
> took a while again for getting all those blockers out of the way. See the 
> previous voting thread [3] for details.
> - Before RC0, an unusually long 2.6.3 release caused 2.7.2 to slip by quite a 
> bit. This release's related discussion threads are linked below: [1] and [2].
>
>Please try the release and vote; the vote will run for the usual 5 days.
>
>Thanks,
>Vinod
>
>[1]: 2.7.2 release plan: http://markmail.org/message/oozq3gvd4nhzsaes 
>
>[2]: Planning Apache Hadoop 2.7.2 http://markmail.org/message/iktqss2qdeykgpqk 
>
>[3]: [VOTE] Release Apache Hadoop 2.7.2 RC0: 
>http://markmail.org/message/5txhvr2qdiqglrwc 
>
>[4] Retracted [VOTE] Release Apache Hadoop 2.7.2 RC1: 
>http://markmail.org/thread/n7ljbsnquihn3wlw

Re: [VOTE] Release Apache Hadoop 2.7.2 RC1

[Arpit Agarwal]

Thanks for putting together this release Vinod.

+1 (binding)

 - Verified signatures
 - Started pseudo-cluster with binary distribution, verified git commit ID
 - Built sources, deployed pseudo-cluster and ran example map reduce jobs, 
DistributedShell, HDFS commands.


PS: Extra file hadoop-2.7.2-RC1-src.tar.gz.md5?





On 12/16/15, 6:49 PM, "Vinod Kumar Vavilapalli"  wrote:

>Hi all,
>
>I've created a release candidate RC1 for Apache Hadoop 2.7.2.
>
>As discussed before, this is the next maintenance release to follow up 2.7.1.
>
>The RC is available for validation at: 
>http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/ 
>
>
>The RC tag in git is: release-2.7.2-RC1
>
>The maven artifacts are available via repository.apache.org 
> at 
>https://repository.apache.org/content/repositories/orgapachehadoop-1026/ 
>
>
>The release-notes are inside the tar-balls at location 
>hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
>this at http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html 
>for quick perusal.
>
>As you may have noted,
> - The RC0 related voting thread got halted due to some critical issues. It 
> took a while again for getting all those blockers out of the way. See the 
> previous voting thread [3] for details.
> - Before RC0, an unusually long 2.6.3 release caused 2.7.2 to slip by quite a 
> bit. This release's related discussion threads are linked below: [1] and [2].
>
>Please try the release and vote; the vote will run for the usual 5 days.
>
>Thanks,
>Vinod
>
>[1]: 2.7.2 release plan: http://markmail.org/message/oozq3gvd4nhzsaes 
>
>[2]: Planning Apache Hadoop 2.7.2 http://markmail.org/message/iktqss2qdeykgpqk 
>
>[3]: [VOTE] Release Apache Hadoop 2.7.2 RC0: 
>http://markmail.org/message/5txhvr2qdiqglrwc
>

Re: [VOTE] Release Apache Hadoop 2.6.3 RC0

[Arpit Agarwal]

+1 (binding)

- Verified signatures for source and binary distributions
- Built jars from source with java 1.7.0_79
- Deployed single-node pseudo-cluster
- Ran example map reduce jobs
- Ran hdfs admin commands, verified NN web UI shows expected usages



On 12/11/15, 4:16 PM, "Junping Du"  wrote:

>
>Hi all developers in hadoop community,
>   I've created a release candidate RC0 for Apache Hadoop 2.6.3 (the next 
> maintenance release to follow up 2.6.2.) according to email thread of release 
> plan 2.6.3 [1]. Sorry for this RC coming a bit late as several blocker issues 
> were getting committed until yesterday. Below is the details:
>
>The RC is available for validation at:
>*http://people.apache.org/~junping_du/hadoop-2.6.3-RC0/
>*
>
>The RC tag in git is: release-2.6.3-RC0
>
>The maven artifacts are staged via repository.apache.org at:
>*https://repository.apache.org/content/repositories/orgapachehadoop-1025/?
>*
>
>You can find my public key at:
>http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>
>Please try the release and vote. The vote will run for the usual 5 days.
>
>Thanks and happy weekend!
>
>
>Cheers,
>
>Junping
>
>
>[1]: 2.6.3 release plan: http://markmail.org/thread/nc2jogbgni37vu6y
>

MR client compatibility across HDP 2.1 - 2.2

[Arpit Agarwal]

Hi yarn-dev,

eBay is asking whether there are any known compatibility issues with MR clients 
using HDP 2.1 libs against 2.2 clusters.

Could one of you please confirm this will be fine?

Thanks,
Arpit

Re: [VOTE] Release Apache Hadoop 2.7.1 RC0

[Arpit Agarwal]

Vinod, thanks for putting together this release.



+1 (binding)

 - Verified signatures
 - Installed binary release on Centos 6 pseudo cluster
* Copied files in and out of HDFS using the shell
* Mounted HDFS via NFS and copied a 10GB file in and out over NFS
* Ran example MapReduce jobs
 - Deployed pseudo cluster from sources on Centos 6, verified
   native bits
 - Deployed pseudo cluster from sources on Windows 2008 R2, verified 
   native bits and ran example MR jobs

Arpit


On 6/29/15, 1:45 AM, "Vinod Kumar Vavilapalli"  wrote:

>Hi all,
>
>I've created a release candidate RC0 for Apache Hadoop 2.7.1.
>
>As discussed before, this is the next stable release to follow up 2.6.0,
>and the first stable one in the 2.7.x line.
>
>The RC is available for validation at:
>*http://people.apache.org/~vinodkv/hadoop-2.7.1-RC0/
>*
>
>The RC tag in git is: release-2.7.1-RC0
>
>The maven artifacts are available via repository.apache.org at
>*https://repository.apache.org/content/repositories/orgapachehadoop-1019/
>*
>
>Please try the release and vote; the vote will run for the usual 5 days.
>
>Thanks,
>Vinod
>
>PS: It took 2 months instead of the planned [1] 2 weeks in getting this
>release out: post-mortem in a separate thread.
>
>[1]: A 2.7.1 release to follow up 2.7.0
>http://markmail.org/thread/zwzze6cqqgwq4rmw

Re: Planning Hadoop 2.6.1 release

[Arpit Agarwal]

HDFS candidates for back-porting to Hadoop 2.6.1. The first two were requested 
in [1].

HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream should be 
non static
HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt 
synchronization

HDFS-7009. Active NN and standby NN have different live nodes.
HDFS-7035. Make adding a new data directory to the DataNode an atomic and 
improve error handling
HDFS-7425. NameNode block deletion logging uses incorrect appender.
HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block 
files are present in the same volume.
HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang datanodes
HDFS-7503. Namenode restart after large deletions can cause slow processReport.
HDFS-7575. Upgrade should generate a unique storage ID for each volume.
HDFS-7579. Improve log reporting during block report rpc failure.
HDFS-7587. Edit log corruption can happen if append fails with a quota 
violation.
HDFS-7596. NameNode should prune dead storages from storageMap.
HDFS-7611. deleteSnapshot and delete of a file can leave orphaned blocks in the 
blocksMap on NameNode restart.
HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause DataNode 
to register successfully with only one NameNode.
HDFS-7733. NFS: readdir/readdirplus return null directory attribute on failure.
HDFS-7831. Fix the starting index and end condition of the loop in 
FileDiffList.findEarlierSnapshotBlocks().
HDFS-7885. Datanode should not trust the generation stamp provided by client.
HDFS-7960. The full block report should prune zombie storages even if they're 
not empty.
HDFS-8072. Reserved RBW space is not released if client terminates while 
writing block.
HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to finalize 
upgrade.


Arpit

[1] Will Hadoop 2.6.1 be released soon? 
http://markmail.org/thread/zlsr6prejyogdyvh



On 4/27/15, 11:47 AM, "Vinod Kumar Vavilapalli"  wrote:

>There were several requests on the user lists [1] for a 2.6.1 release. I
>got many offline comments too.
>
>Planning to do a 2.6.1 release in a few weeks time. We already have a bunch
>of tickets committed to 2.7.1. I created a filter [2] to tracking pending
>tickets.
>
>We need to collectively come up with a list of critical issues. We can use
>the JIRA Target Version field for the same. I see some but not a whole lot
>of new work for this release, most of it is likely going to be pulling in
>critical patches from 2.7.1/2.8 etc.
>
>Thoughts?
>
>Thanks
>+Vinod
>
>[1] Will Hadoop 2.6.1 be released soon?
>http://markmail.org/thread/zlsr6prejyogdyvh
>[2] 2.6.1 pending tickets
>https://issues.apache.org/jira/issues/?filter=12331711
>

Re: [VOTE] Release Apache Hadoop 2.7.0 RC0

[Arpit Agarwal]

+1 for calling 2.7.0 an alpha.

There are a couple of more issues related to incorrect handling of timestamps.

1. HDFS-8163 - Using monotonicNow for block report scheduling causes test 
failures on recently restarted systems
2. HDFS-8179 - DFSClient#getServerDefaults returns null within 1 hour of NN 
start

Tagged both as blockers for 2.7.1.




On 4/17/15, 7:12 AM, "Vinod Kumar Vavilapalli"  wrote:

>Quick look tells me this is a bug that needs fixing.
>
>Am on the road, so couldn't close the vote right after 5 days.
>
>Seeing as this is coming up beyond the voting period, unless you feel strongly 
>against it, I'd like to close the vote as a success but do the following: call 
>this release an alpha for downstream consumption in line with my original 
>proposal, following it up with a 2.7.1 in two weeks.
>
>Thanks
>+Vinod
>
>On Apr 17, 2015, at 2:27 AM, Colin P. McCabe  wrote:
>
>> I would like to fix HDFS-8070, which just came to light.  The impact
>> is that if this isn't fixed, 2.6 clients will be unable to do
>> short-circuit reads against 2.7 datanodes.
>> 
>> best,
>> Colin
>> 
>> On Wed, Apr 15, 2015 at 8:19 PM, Brahma Reddy Battula
>>  wrote:
>>> Need Jcardar changes to support java 7 Byte code..I will work along with 
>>> Todd to get jcardar..
>>> 
>>> Thanks & Regards
>>> Brahma Reddy Battula
>>> 
>>> From: Vinod Kumar Vavilapalli [vino...@hortonworks.com]
>>> Sent: Wednesday, April 15, 2015 6:29 PM
>>> To: common-...@hadoop.apache.org
>>> Subject: Re: [VOTE] Release Apache Hadoop 2.7.0 RC0
>>> 
>>> Tx Brahma. Apologies for missing your offline email.
>>> 
>>> So, did you have luck with an updated JCarder? Or does that need JCarder 
>>> changes that you are waiting on.
>>> 
>>> +Vinod
>>> 
>>> On Apr 15, 2015, at 7:21 AM, Brahma Reddy Battula 
>>>  wrote:
>>> 
 HI Allen
 
 Thanks for updating here [ HDFS-8132 ]..
 
 Jcardar is tied to the Java 6 class format. Hadoop 2.7.0 is our first 
 release that compiled Java 7 class files. Jcarder needs to updated to 
 support Java 7 byte code..
 
 
 I am +1 ( non binding),, after all the regression tests passed against 
 2.7.0-RC0
 
 
 
 
 Thanks & Regards
 Brahma Reddy Battula
 
 From: Allen Wittenauer [a...@altiscale.com]
 Sent: Wednesday, April 15, 2015 12:45 AM
 To: common-...@hadoop.apache.org
 Cc: hdfs-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; 
 mapreduce-...@hadoop.apache.org
 Subject: Re: [VOTE] Release Apache Hadoop 2.7.0 RC0
 
 Someone should look into HDFS-8132, which appears to have been filed 
 against RC0.
 
 On Apr 11, 2015, at 1:44 AM, Vinod Kumar Vavilapalli  
 wrote:
 
> Hi all,
> 
> I've created a release candidate RC0 for Apache Hadoop 2.7.0.
> 
> The RC is available at: 
> http://people.apache.org/~vinodkv/hadoop-2.7.0-RC0/
> 
> The RC tag in git is: release-2.7.0-RC0
> 
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1017/
> 
> As discussed before
> - This release will only work with JDK 1.7 and above
> - I’d like to use this as a starting release for 2.7.x [1], depending on
> how it goes, get it stabilized and potentially use a 2.7.1 in a few
> weeks as the stable release.
> 
> Please try the release and vote; the vote will run for the usual 5 days.
> 
> Thanks,
> Vinod
> 
> [1]: A 2.7.1 release to follow up 2.7.0
> http://markmail.org/thread/zwzze6cqqgwq4rmw
 
>>> 
>

Re: A 2.7.1 release to follow up 2.7.0

[Arpit Agarwal]

+1 for 2.7.1 and +1 for promoting it to 'stable', assuming it includes no new 
features or gratuitous improvements.




Arpit


On 4/9/15, 11:48 AM, "Vinod Kumar Vavilapalli"  wrote:

>Hi all,
>
>I feel like we haven't done a great job of maintaining the previous 2.x
>releases. Seeing as how long 2.7.0 release has taken, I am sure we will
>spend more time stabilizing it, fixing issues etc.
>
>I propose that we immediately follow up 2.7.0 with a 2.7.1 within 2-3
>weeks. The focus obviously is to have blocker issues, bug-fixes and *no*
>features. Improvements are going to be slightly hard to reason about, but I
>propose limiting ourselves to very small improvements, if at all.
>
>The other area of concern with the previous releases had been
>compatibility. With help from Li Lu, I got jdiff reinstated in branch-2
>(though patches are not yet in), and did a pass. In the unavoidable event
>that we find incompatibilities with 2.7.0, we can fix those in 2.7.1 and
>promote that to be the stable release.
>
>Thoughts?
>
>Thanks,+Vinod

Re: Erratic Jenkins behavior

[Arpit Agarwal]

Sorry I don¹t have much insight into this either. Our Jenkins setup is a
bit of a black box to me.

Too few of us have access to the build hosts and it makes debugging
difficult. Perhaps we should grant access to all committers.


On 2/9/15, 11:29 AM, "Colin P. McCabe"  wrote:

>I'm sorry, I don't have any insight into this.  With regard to
>HADOOP-11084, I thought that $BUILD_URL would be unique for each
>concurrent build, which would prevent build artifacts from getting
>mixed up between jobs.  Based on the value of PATCHPROCESS that Kihwal
>posted, perhaps this is not the case?  Perhaps someone can explain how
>this is supposed to work (I am a Jenkins newbie).
>
>regards,
>Colin
>
>On Thu, Feb 5, 2015 at 10:42 AM, Yongjun Zhang 
>wrote:
>> Thanks Kihwal for bringing this up.
>>
>> Seems related to:
>>
>> https://issues.apache.org/jira/browse/HADOOP-11084
>>
>> Hi Andrew/Arpit/Colin/Steve, you guys worked on this jira before, any
>> insight about the issue Kihwal described?
>>
>> Thanks.
>>
>> --Yongjun
>>
>>
>> On Thu, Feb 5, 2015 at 9:49 AM, Kihwal Lee
>>
>> wrote:
>>
>>> I am sure many of us have seen strange jenkins behavior out of the
>>> precommit builds.
>>>
>>> - build artifacts missing
>>> - serving build artifact belonging to another build. This also causes
>>> wrong precommit results to be posted on the bug.
>>> - etc.
>>>
>>> The latest one I saw is disappearance of the unit test stdout/stderr
>>>file
>>> during a build. After a successful run of unit tests, the file
>>>vanished, so
>>> the script could not cat it. It looked like another build process had
>>> deleted it, while this build was in progress.
>>>
>>> It might have something to do with the fact that the patch-dir is set
>>>like
>>> following:
>>>
>>> 
>>>PATCHPROCESS=/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/
>>>../patchprocessI
>>> don't have access to the jenkins build configs or the build machines,
>>>so I
>>> can't debug it further, but I think we need to take care of it sooner
>>>than
>>> later.  Can any one help?
>>>
>>> Kihwal
>>>

Re: Updates on migration to git

[Arpit Agarwal]

I cloned the new repo, built trunk and branch-2, verified all the branches
are present. Also checked a few branches and the recent commit history
matches our existing repo. Everything looks good so far.


On Tue, Aug 26, 2014 at 1:19 PM, Karthik Kambatla 
wrote:

> The git repository is now ready for inspection. I ll take a look shortly,
> but it would be great if a few others could too.
>
> Once we are okay with it, we can ask it to be writable.
>
> On Tuesday, August 26, 2014, Karthik Kambatla  wrote:
>
> > Hi Suresh
> >
> > There was one vote thread on whether to migrate to git, and the
> > implications to the commit process for individual patches and feature
> > branches -
> > https://www.mail-archive.com/common-dev@hadoop.apache.org/msg13447.html
> .
> > Prior to that, there was a discuss thread on the same topic.
> >
> > As INFRA handles the actual migration from subversion to git, the vote
> > didn't include those specifics. The migration is going on as we speak
> (See
> > INFRA-8195). The initial expectation was that the migration would be done
> > in a few hours, but it has been several hours and the last I heard the
> > import was still running.
> >
> > I have elaborated on the points in the vote thread and drafted up a wiki
> > page on how-to-commit -
> https://wiki.apache.org/hadoop/HowToCommitWithGit
> > . We can work on improving this further and call a vote thread on those
> > items if need be.
> >
> > Thanks
> > Karthik
> >
> >
> > On Tue, Aug 26, 2014 at 11:41 AM, Suresh Srinivas <
> sur...@hortonworks.com
> > > wrote:
> >
> >> Karthik,
> >>
> >> I would like to see detailed information on how this migration will be
> >> done, how it will affect the existing project and commit process. This
> >> should be done in a document that can be reviewed instead of in an email
> >> thread on an ad-hoc basis. Was there any voting on this in PMC and
> should
> >> we have a vote to ensure everyone is one the same page on doing this and
> >> how to go about it?
> >>
> >> Regards,
> >> Suresh
> >>
> >>
> >> On Tue, Aug 26, 2014 at 9:17 AM, Karthik Kambatla  >> >
> >> wrote:
> >>
> >> > Last I heard, the import is still going on and appears closer to
> getting
> >> > done. Thanks for your patience with the migration.
> >> >
> >> > I ll update you as and when there is something. Eventually, the git
> repo
> >> > should be at the location in the wiki.
> >> >
> >> >
> >> > On Mon, Aug 25, 2014 at 3:45 PM, Karthik Kambatla  >> >
> >> > wrote:
> >> >
> >> > > Thanks for bringing these points up, Zhijie.
> >> > >
> >> > > By the way, a revised How-to-commit wiki is at:
> >> > > https://wiki.apache.org/hadoop/HowToCommitWithGit . Please feel
> free
> >> to
> >> > > make changes and improve it.
> >> > >
> >> > > On Mon, Aug 25, 2014 at 11:00 AM, Zhijie Shen <
> zs...@hortonworks.com
> >> >
> >> > > wrote:
> >> > >
> >> > >> Do we have any convention about "user.name" and "user.email"? For
> >> > >> example,
> >> > >> we'd like to use @apache.org for the email.
> >> > >>
> >> > >
> >> > > May be, we can ask people to use project-specific configs here and
> use
> >> > > their real name and @apache.org address.
> >> > >
> >> > > Is there any downside to letting people use their global values for
> >> these
> >> > > configs?
> >> > >
> >> > >
> >> > >
> >> > >>
> >> > >> Moreover, do we want to use "--author="Author Name <
> >> em...@address.com >"
> >> > >> when committing on behalf of a particular contributor?
> >> > >>
> >> > >
> >> > > Fetching the email-address is complicated here. Should we use the
> >> > > contributor's email from JIRA? What if that is not their @apache
> >> address?
> >> > >
> >> > >
> >> > >>
> >> > >>
> >> > >> On Mon, Aug 25, 2014 at 9:56 AM, Karthik Kambatla <
> >> ka...@cloudera.com  ');>>
> >> > >> wrote:
> >> > >>
> >> > >> > Thanks for your input, Steve. Sorry for sending the email out
> that
> >> > >> late, I
> >> > >> > sent it as soon as I could.
> >> > >> >
> >> > >> >
> >> > >> > On Mon, Aug 25, 2014 at 2:20 AM, Steve Loughran <
> >> > ste...@hortonworks.com
> >> 
> >> > >> >
> >> > >> > wrote:
> >> > >> >
> >> > >> > > just caught up with this after some offlininess...15:48 PST is
> >> too
> >> > >> late
> >> > >> > for
> >> > >> > > me.
> >> > >> > >
> >> > >> > > I'd be -1 to a change to "master" because of that risk that it
> >> does
> >> > >> break
> >> > >> > > existing code -especially people that have trunk off the git
> >> mirrors
> >> > >> and
> >> > >> > > automated builds/merges to go with it.
> >> > >> > >
> >> > >> >
> >> > >> > Fair enough. It makes sense to leave it as "trunk", unless
> someone
> >> is
> >> > >> > against it being trunk.
> >> > >> >
> >> > >> >
> >> > >> > >
> >> > >> > > "master" may be viewed as the official git way, but it doesn't
> >> have
> >> > to
> >> > >> > be.
> >> > >> > > For git-flow workflows (which we use in slider) master/ is for
> >> > >> releases,
> >> > >> > > develop/ for dev.
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> 

Re: [DISCUSS] Switch to log4j 2

[Arpit Agarwal]

The block state change logs are indeed too noisy at INFO and I've not found
them useful when troubleshooting. Just filed HDFS-6860 to fix that.

This is orthogal to SLF4J migration however moving to SLF4J would help ease
the transition to Log4j 2.

Thanks for the pointer to HDFS-5421 Aaron, looking into it.


On Sat, Aug 16, 2014 at 5:07 AM, Steve Loughran 
wrote:

> On 15 August 2014 17:20, Karthik Kambatla  wrote:
>
> > However, IMO we already log too much at INFO level (particularly YARN).
> > Logging more at DEBUG level and lowering the overhead of enabling DEBUG
> > logging is preferable.
> >
>
> +1
>
> This is the log4j properties file I've adopted for minicluster debugging,
> HDFS is pretty noisy these days too. BlockStateChange, for example. Then
> there's Zookeeper
>
>
>
> log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR
>
> log4j.logger.org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner=WARN
> log4j.logger.org.apache.hadoop.hdfs.server.blockmanagement=WARN
> log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=WARN
> log4j.logger.org.apache.hadoop.hdfs=WARN
> log4j.logger.BlockStateChange=WARN
>
>
> log4j.logger.org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor=WARN
>
> log4j.logger.org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl=WARN
> log4j.logger.org.apache.zookeeper=WARN
> log4j.logger.org.apache.zookeeper.ClientCnxn=FATAL
>
> log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.security=WARN
> log4j.logger.org.apache.hadoop.metrics2=ERROR
> log4j.logger.org.apache.hadoop.util.HostsFileReader=WARN
> log4j.logger.org.apache.hadoop.yarn.event.AsyncDispatcher=WARN
> log4j.logger.org.apache.hadoop.security.token.delegation=WARN
> log4j.logger.org.apache.hadoop.yarn.util.AbstractLivelinessMonitor=WARN
> log4j.logger.org.apache.hadoop.yarn.server.nodemanager.security=WARN
> log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.RMNMInfo=WARN
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[DISCUSS] Switch to log4j 2

[Arpit Agarwal]

I don't recall whether this was discussed before.

I often find our INFO logging to be too sparse for useful diagnosis. A high
performance logging framework will encourage us to log more. Specifically,
Asynchronous Loggers look interesting.
https://logging.apache.org/log4j/2.x/manual/async.html#Performance

What does the community think of switching to log4j 2 in a Hadoop 2.x
release?

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [VOTE] Migration from subversion to git for version control

[Arpit Agarwal]

+1 binding.


On Tue, Aug 12, 2014 at 6:08 AM, Thomas Graves <
tgra...@yahoo-inc.com.invalid> wrote:

> +1 (binding).
>
>
> Tom
>
>
> On Friday, August 8, 2014 9:57 PM, Karthik Kambatla 
> wrote:
>
>
>
> I have put together this proposal based on recent discussion on this topic.
>
> Please vote on the proposal. The vote runs for 7 days.
>
>1. Migrate from subversion to git for version control.
>2. Force-push to be disabled on trunk and branch-* branches. Applying
>changes from any of trunk/branch-* to any of
>  branch-* should be through
>"git cherry-pick -x".
>3. Force-push on feature-branches is allowed. Before pulling in a
>feature, the feature-branch should be rebased on latest trunk and the
>changes applied to trunk through "git rebase --onto" or "git cherry-pick
>".
>4. Every time a feature branch is rebased on trunk, a tag that
>identifies the state before the rebase needs to be created (e.g.
>tag_feature_JIRA-2454_2014-08-07_rebase). These tags can be deleted once
>the feature is pulled into trunk and the tags are no longer useful.
>5. The relevance/use of tags stay the same after the migration.
>
> Thanks
> Karthik
>
> PS: Per Andrew Wang, this should be a "Adoption of New Codebase" kind of
> vote and will be Lazy 2/3 majority of PMC members.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [DISCUSS] Migrate from svn to git for source control?

[Arpit Agarwal]

If by very same workflow you mean a merge-based workflow that would be fine
to call out in the vote proposal.

Separately, do we want to disable force push for version branches
(branch-x) and point release branches (branch-x.y) in addition to trunk?


On Tue, Aug 5, 2014 at 8:18 PM, Alejandro Abdelnur  wrote:

> I would say we can first move to git and keep the very same workflow we
> have today, then we can evolve it.
>
>
> On Tue, Aug 5, 2014 at 6:46 PM, Arpit Agarwal 
> wrote:
>
> > +1 to voting on specific workflow(s).
> >
> >
> > On Tue, Aug 5, 2014 at 5:49 PM, Karthik Kambatla 
> > wrote:
> >
> > > If we are to start a vote thread, will people prefer a vote thread that
> > > includes potential workflows as well?
> > >
> > >
> > > On Tue, Aug 5, 2014 at 5:40 PM, Karthik Kambatla 
> > > wrote:
> > >
> > > > Thanks for your opinions, everyone. Looks like most people are for
> the
> > > > change and no one is against it. Let me start a vote for this.
> > > >
> > > >
> > > > On Mon, Aug 4, 2014 at 4:44 PM, Tsuyoshi OZAWA <
> > ozawa.tsuyo...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > >> Thank you for supplementation, Andrew. Yes, we should go step by
> step
> > > >> and let's discuss review workflows on a another thread.
> > > >>
> > > >> Thanks,
> > > >> - Tsuyoshi
> > > >>
> > > >> On Tue, Aug 5, 2014 at 8:23 AM, Andrew Wang <
> andrew.w...@cloudera.com
> > >
> > > >> wrote:
> > > >> > I think we should take things one step at a time. Switching to git
> > > >> > definitely opens up the possibility for better review workflows,
> but
> > > we
> > > >> can
> > > >> > discuss that on a different thread.
> > > >> >
> > > >> > A few different people have also mentioned Gerrit, so that'd be in
> > the
> > > >> > running along with Github (and I guess ReviewBoard).
> > > >> >
> > > >> > Thanks,
> > > >> > Andrew
> > > >> >
> > > >> >
> > > >> > On Mon, Aug 4, 2014 at 4:17 PM, Tsuyoshi OZAWA <
> > > >> ozawa.tsuyo...@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> >> Thank you for great suggestion, Karthik. +1(non-binding) to use
> > git.
> > > >> >> I'm also using private git repository.
> > > >> >> Additionally, I have one question. Will we accept github-based
> > > >> >> development like Apache Spark? IHMO, it allow us to leverage
> Hadoop
> > > >> >> development, because the cost of sending pull request is very low
> > and
> > > >> >> its review board is great. One concern is that the development
> > > >> >> workflow can change and it can confuse us. What do you think?
> > > >> >>
> > > >> >> Thanks,
> > > >> >> - Tsuyoshi
> > > >> >>
> > > >> >> On Sat, Aug 2, 2014 at 8:43 AM, Karthik Kambatla <
> > ka...@cloudera.com
> > > >
> > > >> >> wrote:
> > > >> >> > Hi folks,
> > > >> >> >
> > > >> >> > From what I hear, a lot of devs use the git mirror for
> > > >> >> development/reviews
> > > >> >> > and use subversion primarily for checking code in. I was
> > wondering
> > > >> if it
> > > >> >> > would make more sense just to move to git. In addition to
> > > subjective
> > > >> >> liking
> > > >> >> > of git, I see the following advantages in our workflow:
> > > >> >> >
> > > >> >> >1. Feature branches - it becomes easier to work on them and
> > keep
> > > >> >> >rebasing against the latest trunk.
> > > >> >> >2. Cherry-picks between branches automatically ensures the
> > exact
> > > >> same
> > > >> >> >commit message and tracks the lineage as well.
> > > >> >> >3. When cutting new branches and/or updating maven versions
> > > etc.,
> > > >> it
> > > >> >> >allows doing all

Re: [DISCUSS] Migrate from svn to git for source control?

[Arpit Agarwal]

+1 to voting on specific workflow(s).


On Tue, Aug 5, 2014 at 5:49 PM, Karthik Kambatla  wrote:

> If we are to start a vote thread, will people prefer a vote thread that
> includes potential workflows as well?
>
>
> On Tue, Aug 5, 2014 at 5:40 PM, Karthik Kambatla 
> wrote:
>
> > Thanks for your opinions, everyone. Looks like most people are for the
> > change and no one is against it. Let me start a vote for this.
> >
> >
> > On Mon, Aug 4, 2014 at 4:44 PM, Tsuyoshi OZAWA  >
> > wrote:
> >
> >> Thank you for supplementation, Andrew. Yes, we should go step by step
> >> and let's discuss review workflows on a another thread.
> >>
> >> Thanks,
> >> - Tsuyoshi
> >>
> >> On Tue, Aug 5, 2014 at 8:23 AM, Andrew Wang 
> >> wrote:
> >> > I think we should take things one step at a time. Switching to git
> >> > definitely opens up the possibility for better review workflows, but
> we
> >> can
> >> > discuss that on a different thread.
> >> >
> >> > A few different people have also mentioned Gerrit, so that'd be in the
> >> > running along with Github (and I guess ReviewBoard).
> >> >
> >> > Thanks,
> >> > Andrew
> >> >
> >> >
> >> > On Mon, Aug 4, 2014 at 4:17 PM, Tsuyoshi OZAWA <
> >> ozawa.tsuyo...@gmail.com>
> >> > wrote:
> >> >
> >> >> Thank you for great suggestion, Karthik. +1(non-binding) to use git.
> >> >> I'm also using private git repository.
> >> >> Additionally, I have one question. Will we accept github-based
> >> >> development like Apache Spark? IHMO, it allow us to leverage Hadoop
> >> >> development, because the cost of sending pull request is very low and
> >> >> its review board is great. One concern is that the development
> >> >> workflow can change and it can confuse us. What do you think?
> >> >>
> >> >> Thanks,
> >> >> - Tsuyoshi
> >> >>
> >> >> On Sat, Aug 2, 2014 at 8:43 AM, Karthik Kambatla  >
> >> >> wrote:
> >> >> > Hi folks,
> >> >> >
> >> >> > From what I hear, a lot of devs use the git mirror for
> >> >> development/reviews
> >> >> > and use subversion primarily for checking code in. I was wondering
> >> if it
> >> >> > would make more sense just to move to git. In addition to
> subjective
> >> >> liking
> >> >> > of git, I see the following advantages in our workflow:
> >> >> >
> >> >> >1. Feature branches - it becomes easier to work on them and keep
> >> >> >rebasing against the latest trunk.
> >> >> >2. Cherry-picks between branches automatically ensures the exact
> >> same
> >> >> >commit message and tracks the lineage as well.
> >> >> >3. When cutting new branches and/or updating maven versions
> etc.,
> >> it
> >> >> >allows doing all the work locally before pushing it to the main
> >> >> branch.
> >> >> >4. Opens us up to potentially using other code-review tools.
> >> (Gerrit?)
> >> >> >5. It is just more convenient.
> >> >> >
> >> >> > I am sure this was brought up before in different capacities. I
> >> believe
> >> >> the
> >> >> > support for git in ASF is healthy now and several downstream
> projects
> >> >> have
> >> >> > moved. Again, from what I hear, ASF INFRA folks make the migration
> >> >> process
> >> >> > fairly easy.
> >> >> >
> >> >> > What do you all think?
> >> >> >
> >> >> > Thanks
> >> >> > Karthik
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> - Tsuyoshi
> >> >>
> >>
> >>
> >>
> >> --
> >> - Tsuyoshi
> >>
> >
> >
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Created] (YARN-2384) Document YARN multihoming settings

[Arpit Agarwal (JIRA)]

Arpit Agarwal created YARN-2384:
---

 Summary: Document YARN multihoming settings
 Key: YARN-2384
 URL: https://issues.apache.org/jira/browse/YARN-2384
 Project: Hadoop YARN
  Issue Type: Bug
  Components: documentation
Affects Versions: 2.6.0
Reporter: Arpit Agarwal


YARN-1994 introduced new settings to improve multihoming support in YARN/MR. 
This Jira is to get the settings documented.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Hadoop Code of Incompatible Changes

[Arpit Agarwal]

I cleared out the wiki page and left a forwarding link to the site docs.
>From a quick scan all the content is included in the site docs.


On Tue, Jul 29, 2014 at 2:14 PM, Sandy Ryza  wrote:

> Eli pointed out to me that this is the up-to-date compatibility guide:
>
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Compatibility.html
>
> Thanks,
> Sandy
>
>
> On Tue, Jul 29, 2014 at 9:39 AM, Sandy Ryza 
> wrote:
>
> > Hi Zhijie,
> >
> > The Hadoop compatibility guide mentions this as "semantic compatibility":
> > http://wiki.apache.org/hadoop/Compatibility
> >
> > My interpretation of the section is that we can't change the behavior of
> > public APIs unless we're fixing buggy behavior.  If the change could
> break
> > an existing application that's behaving reasonably with respect to the
> old
> > API, it's an incompatible change.
> >
> > -Sandy
> >
> >
> >
> > On Tue, Jul 29, 2014 at 9:26 AM, Zhijie Shen 
> > wrote:
> >
> >> Hi folks,
> >>
> >> Recently we have a conversation on YARN-2209 about the incompatible
> >> changes
> >> over releases. For those API changes that will break binary
> compatibility,
> >> source compatibility towards the existing API users, we've already had a
> >> rather clear picture about what we should do. However, YARN-2209 has
> >> introduced another case which I'm not quite sure about, which is kind of
> >> *logic
> >> incompatibility*.
> >>
> >> In detail, an ApplicationMasterProtocol API is going to throw an
> exception
> >> which is not expected before. The exception is a sub-class of
> >> YarnException, such that it doesn't need any method signature change,
> and
> >> won't break any binary/source compatibility. However, the exception is
> not
> >> expected before, but needs to be treated specially at the AM side. Not
> >> being aware of the newly introduced exception, the existing YARN
> >> applications' AM may not handle it exception properly, and is at the
> risk
> >> of being broken on a new YARN cluster after this change.
> >>
> >> An additional thought around this problem is that the change of what
> >> exception is to throw under what situation may be considered as a *soft
> >> *method
> >> signature change, because we're supposed to write this javadoc to tell
> the
> >> users (though we didn't it well in Hadoop), and users refer to it to
> guide
> >> how to handle the exception.
> >>
> >> In a more generalized form, let's assume we have a method, which behaves
> >> as
> >> A, in release 1.0. However, in release 2.0, the method signature has
> kept
> >> the same, while its behavior is altered from A to B. A and B are
> different
> >> behaviors. In this case, do we consider it as an incompatible change?
> >>
> >> I think it's somewhat a common issue, such that I raise it on the
> mailing
> >> list. Please share your ideas.
> >>
> >> Thanks,
> >> Zhijie
> >>
> >> --
> >> Zhijie Shen
> >> Hortonworks Inc.
> >> http://hortonworks.com/
> >>
> >> --
> >> CONFIDENTIALITY NOTICE
> >> NOTICE: This message is intended for the use of the individual or entity
> >> to
> >> which it is addressed and may contain information that is confidential,
> >> privileged and exempt from disclosure under applicable law. If the
> reader
> >> of this message is not the intended recipient, you are hereby notified
> >> that
> >> any printing, copying, dissemination, distribution, disclosure or
> >> forwarding of this communication is strictly prohibited. If you have
> >> received this communication in error, please contact the sender
> >> immediately
> >> and delete it from your system. Thank You.
> >>
> >
> >
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [VOTE] Change by-laws on release votes: 5 days instead of 7

[Arpit Agarwal]

+1 Arpit


On Tue, Jun 24, 2014 at 1:53 AM, Arun C Murthy  wrote:

> Folks,
>
>  As discussed, I'd like to call a vote on changing our by-laws to change
> release votes from 7 days to 5.
>
>  I've attached the change to by-laws I'm proposing.
>
>  Please vote, the vote will the usual period of 7 days.
>
> thanks,
> Arun
>
> 
>
> [main]$ svn diff
> Index: author/src/documentation/content/xdocs/bylaws.xml
> ===
> --- author/src/documentation/content/xdocs/bylaws.xml   (revision 1605015)
> +++ author/src/documentation/content/xdocs/bylaws.xml   (working copy)
> @@ -344,7 +344,16 @@
>  Votes are open for a period of 7 days to allow all active
>  voters time to consider the vote. Votes relating to code
>  changes are not subject to a strict timetable but should be
> -made as timely as possible.
> +made as timely as possible.
> +
> + 
> +  Product Release - Vote Timeframe
> +   Release votes, alone, run for a period of 5 days. All other
> + votes are subject to the above timeframe of 7 days.
> + 
> +   
> +   
> +
> 
> 
>  
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Created] (YARN-1994) Expose YARN/MR endpoints on multiple interfaces

[Arpit Agarwal (JIRA)]

Arpit Agarwal created YARN-1994:
---

 Summary: Expose YARN/MR endpoints on multiple interfaces
 Key: YARN-1994
 URL: https://issues.apache.org/jira/browse/YARN-1994
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: nodemanager, resourcemanager, webapp
Affects Versions: 2.4.0
Reporter: Arpit Agarwal


YARN and MapReduce daemons currently do not support specifying a wildcard 
address for the server endpoints. This prevents the endpoints from being 
accessible from all interfaces on a multihomed machine.

A preliminary shows the following candidates:
#  yarn.nodemanager.address
#  yarn.nodemanager.webapp.address
#  yarn.nodemanager.webapp.https.address
#  yarn.resourcemanager.address
#  yarn.resourcemanager.webapp.address
#  yarn.resourcemanager.webapp.https.address
#  yarn.resourcemanager.resource-tracker.address
#  yarn.resourcemanager.scheduler.address
#  yarn.resourcemanager.admin.address
#  mapreduce.jobhistory.address

Note that if we do specify INADDR_ANY for any of the options, it will break 
clients as they will attempt to connect to 0.0.0.0. We need a solution that 
allows specifying a hostname or IP-address for clients while requesting 
wildcard bind for the servers.
#  mapreduce.jobhistory.admin.address
#  mapreduce.jobhistory.webapp.address
#  mapreduce.jobhistory.webapp.https.address
#  mapreduce.history.server.http.address (Deprecated)
#  yarn.timeline-service.webapp.address
#  yarn.timeline-service.webapp.https.address




--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Next releases

[Arpit Agarwal]

This merge to branch-2 is complete. The changes have been merged to
branch-2 and target version set to 2.4.0 (r1556076).


On Fri, Jan 3, 2014 at 4:13 PM, Arpit Agarwal wrote:

> We plan to merge HDFS-2832 to branch-2 next week for inclusion in 2.4.
>
>
>
> On Fri, Dec 6, 2013 at 1:53 PM, Arun C Murthy  wrote:
>
>> Thanks Suresh & Colin.
>>
>> Please update the Roadmap wiki with your proposals.
>>
>> As always, we will try our best to get these in - but we can collectively
>> decide to slip some of these to subsequent releases based on timelines.
>>
>> Arun
>>
>> On Dec 6, 2013, at 10:43 AM, Suresh Srinivas 
>> wrote:
>>
>> > Arun,
>> >
>> > I propose the following changes for 2.3:
>> > - There have been a lot of improvements related to supporting http
>> policy.
>> > - There is a still discussion going on, but I would like to deprecate
>> > BackupNode in 2.3 as well.
>> > - We are currently working on rolling upgrades related change in HDFS.
>> We
>> > might add a couple of changes that enables rolling upgrades from 2.3
>> > onwards (hopefully we can this done by December)
>> >
>> > I propose the following for 2.4 release, if they are tested and stable:
>> > - Heterogeneous storage support - HDFS-2832
>> > - Datanode cache related change - HDFS-4949
>> > - HDFS ACLs - HDFS-4685
>> > - Rolling upgrade changes
>> >
>> > Let me know if you want me to update the wiki.
>> >
>> > Regards,
>> > Suresh
>> >
>>
>> On Dec 6, 2013, at 12:27 PM, Colin McCabe  wrote:
>>
>> > If 2.4 is released in January, I think it's very unlikely to include
>> > symlinks.  There is still a lot of work to be done before they're
>> > usable.  You can look at the progress on HADOOP-10019.  For some of
>> > the subtasks, it will require some community discussion before any
>> > code can be written.
>> >
>> > For better or worse, symlinks have not been requested by users as
>> > often as features like NFS export, HDFS caching, ACLs, etc, so effort
>> > has been focused on those instead.
>> >
>> > For now, I think we should put the symlinks-disabling patches
>> > (HADOOP-10020, etc) into branch-2, so that they will be part of the
>> > next releases without additional effort.
>> >
>> > I would like to see HDFS caching make it into 2.4.  The APIs and
>> > implementation are beginning to stabilize, and around January it
>> > should be ok to backport to a stable branch.
>> >
>> > best,
>> > Colin
>> >
>> >
>> > On Thu, Nov 7, 2013 at 6:42 PM, Arun C Murthy 
>> wrote:
>> >
>> >> Gang,
>> >>
>> >> Thinking through the next couple of releases here, appreciate f/b.
>> >>
>> >> # hadoop-2.2.1
>> >>
>> >> I was looking through commit logs and there is a *lot* of content here
>> >> (81 commits as on 11/7). Some are features/improvements and some are
>> fixes
>> >> - it's really hard to distinguish what is important and what isn't.
>> >>
>> >> I propose we start with a blank slate (i.e. blow away branch-2.2 and
>> >> start fresh from a copy of branch-2.2.0)  and then be very careful and
>> >> meticulous about including only *blocker* fixes in branch-2.2. So,
>> most of
>> >> the content here comes via the next minor release (i.e. hadoop-2.3)
>> >>
>> >> In future, we continue to be *very* parsimonious about what gets into a
>> >> patch release (major.minor.patch) - in general, these should be only
>> >> *blocker* fixes or key operational issues.
>> >>
>> >> # hadoop-2.3
>> >>
>> >> I'd like to propose the following features for YARN/MR to make it into
>> >> hadoop-2.3 and punt the rest to hadoop-2.4 and beyond:
>> >> * Application History Server - This is happening in  a branch and is
>> >> close; with it we can provide a reasonable experience for new
>> frameworks
>> >> being built on top of YARN.
>> >> * Bug-fixes in RM Restart
>> >> * Minimal support for long-running applications (e.g. security) via
>> >> YARN-896
>> >> * RM Fail-over via ZKFC
>> >> * Anything else?
>> >>
>> >> HDFS???
>> >>
>> >> Overall, I feel like we have a decent chance of rolling hadoop-2.3 by
&

Re: [Vote] Merge branch-trunk-win to trunk

[Arpit Agarwal]

+1 non-binding.

I have extensively tested this on both Windows and Linux over the last few
months.

Thanks,
-Arpit

On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins  wrote:

> Bobby raises some good questions.  A related one, since most current
> developers won't add Windows support for new features that are
> platform specific is it assumed that Windows development will either
> lag or will people actively work on keeping Windows up with the
> latest?  And vice versa in case Windows support is implemented first.
>
> Is there a jira for resolving the outstanding TODOs in the code base
> (similar to HDFS-2148)?  Looks like this merge doesn't introduce many
> which is great (just did a quick diff and grep).
>
> Thanks,
> Eli
>
> On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans  wrote:
> > After this is merged in is Windows still going to be a second class
> > citizen but happens to work for more than just development or is it a
> > fully supported platform where if something breaks it can block a
> release?
> >  How do we as a community intend to keep Windows support from breaking?
> > We don't have any Jenkins slaves to be able to run nightly tests to
> > validate everything still compiles/runs.  This is not a blocker for me
> > because we often rely on individuals and groups to test Hadoop, but I do
> > think we need to have this discussion before we put it in.
> >
> > --Bobby
> >
> > On 2/26/13 4:55 PM, "Suresh Srinivas"  wrote:
> >
> >>I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
> >>I
> >>am happy to announce that we are ready for the merge.
> >>
> >>Here is a brief recap on the highlights of the work done:
> >>- Command-line scripts for the Hadoop surface area
> >>- Mapping the HDFS permissions model to Windows
> >>- Abstracted and reconciled mismatches around differences in Path
> >>semantics
> >>in Java and Windows
> >>- Native Task Controller for Windows
> >>- Implementation of a Block Placement Policy to support cloud
> >>environments,
> >>more specifically Azure.
> >>- Implementation of Hadoop native libraries for Windows (compression
> >>codecs, native I/O)
> >>- Several reliability issues, including race-conditions, intermittent
> test
> >>failures, resource leaks.
> >>- Several new unit test cases written for the above changes
> >>
> >>Please find the details of the work in CHANGES.branch-trunk-win.txt -
> >>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9
> >,
> >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
> >>ported from branch-1-win to a branch based on trunk.
> >>
> >>For details of the testing done, please see the thread -
> >>http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
> >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> >>
> >>This was a large undertaking that involved developing code, testing the
> >>entire Hadoop stack, including scale tests. This is made possible only
> >>with
> >>the contribution from many many folks in the community. Following people
> >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
> >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> Sumadhur
> >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
> >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
> >>Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze,
> Suresh
> >>Srinivas and Sanjay Radia. There are many others who contributed as well
> >>providing feedback and comments on numerous jiras.
> >>
> >>The vote will run for seven days and will end on March 5, 6:00PM PST.
> >>
> >>Regards,
> >>Suresh
> >>
> >>
> >>
> >>
> >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> >>wrote:
> >>
> >>> It is super exciting to look at the prospect of these changes being
> >>>merged
> >>> to trunk. Having Windows as one of the supported Hadoop platforms is a
> >>> fantastic opportunity both for the Hadoop project and Microsoft
> >>>customers.
> >>>
> >>> This work began around a year back when a few of us started with a
> basic
> >>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft
> have
> >>> made significant progress in the follow