Re: [VOTE] Merge yarn-native-services branch into trunk

[Sunil G]

+1 (binding)

Tested the branch code and brought up services like sleep and httpd. Also
verified UI as well.

- Sunil


On Tue, Oct 31, 2017 at 1:36 AM Jian He  wrote:

> Hi All,
>
> I would like to restart the vote for merging yarn-native-services to trunk.
> Since last vote, we have been working on several issues in documentation,
> DNS, CLI modifications etc. We believe now the feature is in a much better
> shape.
>
> Some back ground:
> At a high level, the following are the key feautres implemented.
> - YARN-5079[1]. A native YARN framework (ApplicationMaster) to orchestrate
> existing services to YARN either docker or non-docker based.
> - YARN-4793[2]. A Rest API service embeded in RM (optional)  for user to
> deploy a service via a simple JSON spec
> - YARN-4757[3]. Extending today's service registry with a simple DNS
> service to enable users to discover services deployed on YARN via standard
> DNS lookup
> - YARN-6419[4]. UI support for native-services on the new YARN UI
> All these new services are optional and are sitting outside of the
> existing system, and have no impact on existing system if disabled.
>
> Special thanks to a team of folks who worked hard towards this: Billie
> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma
> K S, Sunil G, Akhil PB, Eric Yang. This effort could not be possible
> without their ideas and hard work.
> Also thanks Allen for some review and verifications.
>
> Thanks,
> Jian
>
> [1] https://issues.apache.org/jira/browse/YARN-5079
> [2] https://issues.apache.org/jira/browse/YARN-4793
> [3] https://issues.apache.org/jira/browse/YARN-4757
> [4] https://issues.apache.org/jira/browse/YARN-6419
>

[jira] [Created] (HDFS-12779) [READ] Allow cluster id to be specified to the Image generation tool

[Virajith Jalaparti (JIRA)]

Virajith Jalaparti created HDFS-12779:
-

 Summary: [READ] Allow cluster id to be specified to the Image 
generation tool
 Key: HDFS-12779
 URL: https://issues.apache.org/jira/browse/HDFS-12779
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Virajith Jalaparti






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Merge yarn-native-services branch into trunk

[Eric Yang]

+1 (binding) for branch merge.  I couldn’t get the patch to work properly 
though.  Yarn-native-services branch works fine.
If bugs surface in trunk, we will fix it.

Regards,
Eric

On 11/3/17, 2:45 PM, "Wangda Tan"  wrote:

Thanks all for the great works!

+1 (Binding). Tried to use native services to build and run applications
successfully.

- Wangda

On Fri, Nov 3, 2017 at 11:50 AM, Arun Suresh  wrote:

> +1 (binding)
>
> Cheers
> -Arun
>
> On Nov 3, 2017 11:44 AM, "Chandni Singh"  wrote:
>
> > +1
> > Thanks,
> > Chandni
> > 
> > From: Jian He 
> > Sent: Monday, October 30, 2017 1:49 PM
> > To: yarn-...@hadoop.apache.org; Hdfs-dev; Hadoop Common;
> > mapreduce-...@hadoop.apache.org
> > Subject: Re: [VOTE] Merge yarn-native-services branch into trunk
> >
> > Few more things:
> >
> > This is the document for trying a non-docker service on YARN.
> > https://github.com/apache/hadoop/blob/yarn-native-
> > services/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/
> > src/site/markdown/yarn-service/QuickStart.md
> >
> > And the document for a docker based service
> > https://github.com/apache/hadoop/blob/yarn-native-
> > services/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/
> > src/site/markdown/yarn-service/Examples.md
> >
> > And the vote lasts 7 days as usual.
> >
> > Thanks,
> > Jian
> >
> > On Oct 30, 2017, at 1:06 PM, Jian He  > e...@hortonworks.com>> wrote:
> >
> > Hi All,
> >
> > I would like to restart the vote for merging yarn-native-services to
> trunk.
> > Since last vote, we have been working on several issues in 
documentation,
> > DNS, CLI modifications etc. We believe now the feature is in a much
> better
> > shape.
> >
> > Some back ground:
> > At a high level, the following are the key feautres implemented.
> > - YARN-5079[1]. A native YARN framework (ApplicationMaster) to
> orchestrate
> > existing services to YARN either docker or non-docker based.
> > - YARN-4793[2]. A Rest API service embeded in RM (optional)  for user to
> > deploy a service via a simple JSON spec
> > - YARN-4757[3]. Extending today's service registry with a simple DNS
> > service to enable users to discover services deployed on YARN via
> standard
> > DNS lookup
> > - YARN-6419[4]. UI support for native-services on the new YARN UI
> > All these new services are optional and are sitting outside of the
> > existing system, and have no impact on existing system if disabled.
> >
> > Special thanks to a team of folks who worked hard towards this: Billie
> > Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith
> Sharma
> > K S, Sunil G, Akhil PB, Eric Yang. This effort could not be possible
> > without their ideas and hard work.
> > Also thanks Allen for some review and verifications.
> >
> > Thanks,
> > Jian
> >
> > [1] https://issues.apache.org/jira/browse/YARN-5079
> > [2] https://issues.apache.org/jira/browse/YARN-4793
> > [3] https://issues.apache.org/jira/browse/YARN-4757
> > [4] https://issues.apache.org/jira/browse/YARN-6419
> >
> >
> >
> >
> > -
> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >
> >
>

[DISCUSS] A final minor release off branch-2?

[Vinod Kumar Vavilapalli]

Hi all,

With 3.0.0 GA around the corner (tx for the push, Andrew!), 2.9.0 RC out (tx 
Arun / Subru!) and 2.8.2 (tx Junping!), I think it's high time we have a 
discussion on how we manage our developmental bandwidth between 2.x line and 
3.x lines.

Once 3.0 GA goes out, we will have two parallel and major release lines. The 
last time we were in this situation was back when we did 1.x -> 2.x jump.

The parallel releases implies overhead of decisions, branch-merges and 
back-ports. Right now we already do backports for 2.7.5, 2.8.2, 2.9.1, 3.0.1 
and potentially a 3.1.0 in a few months after 3.0.0 GA. And many of these lines 
- for e.g 2.8, 2.9 - are going to be used for a while at a bunch of large 
sites! At the same time, our users won't migrate to 3.0 GA overnight - so we do 
have to support two parallel lines.

I propose we start thinking of the fate of branch-2. The idea is to have one 
final release that helps our users migrate from 2.x to 3.x. This includes any 
changes on the older line to bridge compatibility issues, upgrade issues, 
layout changes, tooling etc.

We have a few options I think
 (A)
-- Make 2.9.x the last minor release off branch-2
-- Have a maintenance release that bridges 2.9 to 3.x
-- Continue to make more maintenance releases on 2.8 and 2.9 as necessary
-- All new features obviously only go into the 3.x line as no features can 
go into the maint line.

 (B)
-- Create a new 2.10 release which doesn't have any new features, but as a 
bridging release
-- Continue to make more maintenance releases on 2.8, 2.9 and 2.10 as 
necessary
-- All new features, other than the bridging changes, go into the 3.x line

 (C)
-- Continue making branch-2 releases and postpone this discussion for later

I'm leaning towards (A) or to a lesser extent (B). Willing to hear otherwise.

Now, this obviously doesn't mean blocking of any more minor releases on 
branch-2. Obviously, any interested committer / PMC can roll up his/her 
sleeves, create a release plan and release, but we all need to acknowledge that 
versions are not cheap and figure out how the community bandwidth is split 
overall.

Thanks
+Vinod
PS: The proposal is obviously not to force everyone to go in one direction but 
more of a nudging the community to figure out if we can focus a major part of 
of our bandwidth on one line. I had a similar concern when we were doing 2.8 
and 3.0 in parallel, but the impending possibility of spreading too thin is 
much worse IMO.
PPS: (C) is a bad choice. With 2.8 and 2.9 we are already seeing user adoption 
splintering between two lines. With 2.10, 2.11 etc coexisting with 3.0, 3.1 
etc, we will revisit the mad phase years ago when we had 0.20.x, 0.20-security 
coexisting with 0.21, 0.22 etc.

[jira] [Created] (HDFS-12778) [READ] Report multiple locations for PROVIDED blocks

[Virajith Jalaparti (JIRA)]

Virajith Jalaparti created HDFS-12778:
-

 Summary: [READ] Report multiple locations for PROVIDED blocks
 Key: HDFS-12778
 URL: https://issues.apache.org/jira/browse/HDFS-12778
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Virajith Jalaparti
Priority: Major


On {{getBlockLocations}}, only one Datanode is returned as the location for all 
PROVIDED blocks. This can hurt the performance of applications which typically 
3 locations per block. We need to return multiple Datanodes for each PROVIDED 
block for better application performance/resilience. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12777) [READ] Share URI prefix across Provided blocks to lower memory footprint.

[Virajith Jalaparti (JIRA)]

Virajith Jalaparti created HDFS-12777:
-

 Summary: [READ] Share URI prefix across Provided blocks to lower 
memory footprint.
 Key: HDFS-12777
 URL: https://issues.apache.org/jira/browse/HDFS-12777
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Virajith Jalaparti
Priority: Major


As opposed to local blocks, each DN keeps track of all blocks in PROVIDED 
storage. This can be millions of blocks for 100s of TBs of PROVIDED data. This 
JIRA aims to reduce the memory footprint of these blocks by using a common URI 
prefix across all PROVIDED blocks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12776) [READ] Increasing replication for PROVIDED files should create local replicas

[Virajith Jalaparti (JIRA)]

Virajith Jalaparti created HDFS-12776:
-

 Summary: [READ] Increasing replication for PROVIDED files should 
create local replicas
 Key: HDFS-12776
 URL: https://issues.apache.org/jira/browse/HDFS-12776
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Virajith Jalaparti
Priority: Major


For PROVIDED files, set replication only works when the target datanode does 
not have a PROVIDED volume. In a cluster, where all Datanodes have PROVIDED 
volumes, set replication does not work.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12775) [READ] Fix capacity reporting for Provided volumes

[Virajith Jalaparti (JIRA)]

Virajith Jalaparti created HDFS-12775:
-

 Summary: [READ] Fix capacity reporting for Provided volumes
 Key: HDFS-12775
 URL: https://issues.apache.org/jira/browse/HDFS-12775
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Virajith Jalaparti
Priority: Major


Provided Volumes currently report infinite capacity and 0 space used. This JIRA 
aims to replace this with a capacity report that users would expect.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC0)

[Arun Suresh]

Hey Vinod,

I've cleaned up the RC directory as you requested.

Cheers
-Arun

On Fri, Nov 3, 2017 at 4:09 PM, Vinod Kumar Vavilapalli 
wrote:

> Arun / Subru,
>
> Thanks for the great work!
>
> Few quick comments
>  - Can you cleanup the RC folder to only have tar.gz and src.tar.gz and
> their signatures and delete everything else? So that it's easy to pick up
> the important bits for the voters. For e.g, like this
> http://people.apache.org/~vinodkv/hadoop-2.8.1-RC3/
>  - Can you put the generated CHANGES.html and releasenotes.html instead of
> the md files, for quicker perusal?
>
> Thanks
> +Vinod
>
> On Nov 3, 2017, at 3:50 PM, Arun Suresh  wrote:
>
> Hi folks,
>
> Apache Hadoop 2.9.0 is the first stable release of Hadoop 2.9 line and
> will be the latest stable/production release for Apache Hadoop - it
> includes 30 New Features with 500+ subtasks, 407 Improvements, 787 Bug
> fixes new fixed issues since 2.8.2 .
>
>  More information about the 2.9.0 release plan can be found here:
> *https://cwiki.apache.org/confluence/display/HADOOP/
> Roadmap#Roadmap-Version2.9
>  Roadmap#Roadmap-Version2.9>*
>
>  New RC is available at:
> http://home.apache.org/~asuresh/hadoop-2.9.0-RC0/
>
>  The RC tag in git is: release-2.9.0-RC0, and the latest commit id is:
> 6697f0c18b12f1bdb99cbdf81394091f4fef1f0a
>
>  The maven artifacts are available via repository.apache.org at:
> *https://repository.apache.org/content/repositories/orgapachehadoop-1065/
>  >*
>
>  Please try the release and vote; the vote will run for the usual 5
> days, ending on 11/10/2017 4pm PST time.
>
> Thanks,
>
> Arun/Subru
>
>
>

Re: Cutting branch-2.9

[Subru Krishnan]

Hello Junping

Thanks for taking the time/effort to go through this. Including our reply
so that folks in the list have the full context.

> 1. It looks like we change from log4j to sel4j (HADOOP-12956) in 2.9
which is a huge incompatible change. Do we have consensus on this in
community (as I saw HBase community are deny this kind of change in minor
release)? If so, we should document somewhere what is changed and what's
the impact.
As per the compatibility guidelines (http://hadoop.apache.org/
docs/current/hadoop-project-dist/hadoop-common/Compatibility.html#Log4j_
Configuration_Files), our understanding is that if SLF4j can read the old
Log4j property files, we should be good - and it is the case.

>  2. Some APIs (deprecated in 2.8) get removed. My understanding is we
only remove deprecated API in next major release (3.0) but not minor
release. Please double check this with some documents or other guys for
certain answer.
We see only 2 DEPRECATED methods have been removed:
- The AllocateResponse.newInstance method and the RMProxy.createRMProxy
methods. Both are internal builders and are not really APIs IMHO - so we
feel this should not be a blocker.

>  3. Some stable APIs get removed, like: YarnScheduler.allocate(), etc. We
may need to add it back before we go for RC stage.
Are you sure you are running JACC against latest 2.8.2 artifacts - as we
saw a much smaller delta when we ran jdiff ? Case in point: Looks like the
YarnScheduler#allocate() changes were already part of 2.8.2 as per (
https://github.com/apache/hadoop/blob/branch-2.8.2/
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-
resourcemanager/src/main/java/org/apache/hadoop/yarn/server/
resourcemanager/scheduler/YarnScheduler.java) and the relevant JIRA is
YARN-5221.

-Subru/Arun

On Fri, Nov 3, 2017 at 4:24 PM, Junping Du  wrote:

> Below is some findings from my recently run of JACC (
> https://builds.apache.org/job/Hadoop-2.9-JACC/12/
> artifact/target/compat-check/report.html) job against 2.9 and 2.8.2. I
> have discussed with Arun and Subru offline on jdiff report who convinced me
> some of items are not a big concern. Just tried to list here in case people
> in community have different voices:
>
>  My original concerns of incompatibilities:
>  1. It looks like we change from log4j to sel4j (HADOOP-12956) in 2.9
> which is a huge incompatible change. Do we have consensus on this in
> community - as I saw HBase community are deny this kind of change in minor
> release? I talked with several other guys who seems to have different
> opinion.
>
> 2. Some APIs (deprecated in 2.8) get removed. My understanding is we
> only remove deprecated API in next major release (3.0) but not minor
> release. - Arun and Subru convinced me that removed 2 methods are not
> supposed to be used by other downstream projects
>
> 3. Some stable APIs get removed, like: YarnScheduler.allocate(), etc.
> We may need to add it back for compatibility of Scheduler API.
>
> Thoughts?
>
> Thanks,
>
> Junping
>
> 
> From: Arun Suresh 
> Sent: Thursday, November 2, 2017 11:55 PM
> To: 俊平堵
> Cc: Arun Suresh; su...@apache.org; common-...@hadoop.apache.org;
> yarn-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> mapreduce-...@hadoop.apache.org
> Subject: Re: Cutting branch-2.9
>
> Quick update folks.
>
> We just cut branch-2.9.0, and set the mvn version to 2.9.0
> We've also updated the version on branch-2.9 to 2.9.1-SNAPSHOT.
>
> Expect an RC in the next 24 hrs :)
>
> Cheers
> -Arun/Subru
>
> On Tue, Oct 31, 2017 at 5:24 PM, Arun Suresh 
> wrote:
>
> > Hi Junping,
> >
> > > In the mean time, branch-2.9 should be reserved for next 2.9 point
> > release (2.9.1) and branch-2 should be reserved for next minor release
> > (2.10.0 or whatever name it is). Thoughts?
> > Yup, That is the current plan. We've updated mvn versions in branch-2 to
> > point to 2.10.0-SNAPSHOT.
> > Once we cut branch-2.9.0, we will update mvn versions on branch-2.9.0 to
> > 2.9.0 and set versions on branch-2.9 to 2.9.1-SNAPSHOT (We plan to do
> that
> > by Thursday)
> >
> > Cheers
> > -Arun
> >
> > On Tue, Oct 31, 2017 at 2:35 PM, 俊平堵  wrote:
> >
> >> Hi Arun/Subru,
> >> Thanks for updating on 2.9.0 release progress. A quick question
> here:
> >> are we planning to release from branch-2.9 directly?
> >> I doubt this as it seems against our current branch release
> practice (
> >> https://wiki.apache.org/hadoop/HowToRelease#Branching). To get rid of
> >> any confusion, I would suggest to cut-off branch-2.9.0 for 2.9.0 release
> >> work. In the mean time, branch-2.9 should be reserved for next 2.9 point
> >> release (2.9.1) and branch-2 should be reserved for next minor release
> >> (2.10.0 or whatever name it is). Thoughts?
> >>
> >> bq. @Junping, lets move the jdiff conversation to separate thread.
> >> Sure. I will reply jdiff in 

Re: Cutting branch-2.9

[Junping Du]

Below is some findings from my recently run of JACC 
(https://builds.apache.org/job/Hadoop-2.9-JACC/12/artifact/target/compat-check/report.html)
 job against 2.9 and 2.8.2. I have discussed with Arun and Subru offline on 
jdiff report who convinced me some of items are not a big concern. Just tried 
to list here in case people in community have different voices:

 My original concerns of incompatibilities:
 1. It looks like we change from log4j to sel4j (HADOOP-12956) in 2.9 which 
is a huge incompatible change. Do we have consensus on this in community - as I 
saw HBase community are deny this kind of change in minor release? I talked 
with several other guys who seems to have different opinion.

2. Some APIs (deprecated in 2.8) get removed. My understanding is we only 
remove deprecated API in next major release (3.0) but not minor release. - Arun 
and Subru convinced me that removed 2 methods are not supposed to be used by 
other downstream projects

3. Some stable APIs get removed, like: YarnScheduler.allocate(), etc. We 
may need to add it back for compatibility of Scheduler API.

Thoughts?

Thanks,

Junping


From: Arun Suresh 
Sent: Thursday, November 2, 2017 11:55 PM
To: 俊平堵
Cc: Arun Suresh; su...@apache.org; common-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org
Subject: Re: Cutting branch-2.9

Quick update folks.

We just cut branch-2.9.0, and set the mvn version to 2.9.0
We've also updated the version on branch-2.9 to 2.9.1-SNAPSHOT.

Expect an RC in the next 24 hrs :)

Cheers
-Arun/Subru

On Tue, Oct 31, 2017 at 5:24 PM, Arun Suresh  wrote:

> Hi Junping,
>
> > In the mean time, branch-2.9 should be reserved for next 2.9 point
> release (2.9.1) and branch-2 should be reserved for next minor release
> (2.10.0 or whatever name it is). Thoughts?
> Yup, That is the current plan. We've updated mvn versions in branch-2 to
> point to 2.10.0-SNAPSHOT.
> Once we cut branch-2.9.0, we will update mvn versions on branch-2.9.0 to
> 2.9.0 and set versions on branch-2.9 to 2.9.1-SNAPSHOT (We plan to do that
> by Thursday)
>
> Cheers
> -Arun
>
> On Tue, Oct 31, 2017 at 2:35 PM, 俊平堵  wrote:
>
>> Hi Arun/Subru,
>> Thanks for updating on 2.9.0 release progress. A quick question here:
>> are we planning to release from branch-2.9 directly?
>> I doubt this as it seems against our current branch release practice (
>> https://wiki.apache.org/hadoop/HowToRelease#Branching). To get rid of
>> any confusion, I would suggest to cut-off branch-2.9.0 for 2.9.0 release
>> work. In the mean time, branch-2.9 should be reserved for next 2.9 point
>> release (2.9.1) and branch-2 should be reserved for next minor release
>> (2.10.0 or whatever name it is). Thoughts?
>>
>> bq. @Junping, lets move the jdiff conversation to separate thread.
>> Sure. I will reply jdiff in separated thread.
>>
>> Thanks,
>>
>> Junping
>>
>> 2017-10-31 13:44 GMT-07:00 Arun Suresh :
>>
>>> Hello folks
>>>
>>> We just cut branch-2.9 since all the critical/blocker issues are now
>>> resolved.
>>> We plan to perform some sanity checks for the rest of the week and cut
>>> branch-2.9.0 and push out an RC0 by the end of the week.
>>>
>>> Kindly refrain from committing to branch-2.9 without giving us a heads
>>> up.
>>>
>>> @Junping, lets move the jdiff conversation to separate thread.
>>>
>>> Cheers
>>> -Arun/Subru
>>>
>>> On Mon, Oct 30, 2017 at 12:39 PM, Subru Krishnan 
>>> wrote:
>>>
>>> > We want to give heads up that we are going to cut branch-2.9 tomorrow
>>> > morning.
>>> >
>>> > We are down to 3 blockers and they all are close to being committed
>>> > (thanks everyone):
>>> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.9+Release
>>> >
>>> > There are 4 other non-blocker JIRAs that are targeted for 2.9.0 which
>>> are
>>> > close to completion.
>>> >
>>> > Folks who are working/reviewing these, kindly prioritize accordingly so
>>> > that we can make the release on time.
>>> > https://issues.apache.org/jira/browse/YARN-7398?filter=12342468
>>> >
>>> > Thanks in advance!
>>> >
>>> > -Subru/Arun
>>> >
>>> >
>>> >
>>>
>>
>>
>


-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC0)

[Vinod Kumar Vavilapalli]

Arun / Subru,

Thanks for the great work!

Few quick comments
 - Can you cleanup the RC folder to only have tar.gz and src.tar.gz and their 
signatures and delete everything else? So that it's easy to pick up the 
important bits for the voters. For e.g, like this 
http://people.apache.org/~vinodkv/hadoop-2.8.1-RC3/ 

 - Can you put the generated CHANGES.html and releasenotes.html instead of the 
md files, for quicker perusal?

Thanks
+Vinod

> On Nov 3, 2017, at 3:50 PM, Arun Suresh  wrote:
> 
> Hi folks,
> 
> Apache Hadoop 2.9.0 is the first stable release of Hadoop 2.9 line and
> will be the latest stable/production release for Apache Hadoop - it
> includes 30 New Features with 500+ subtasks, 407 Improvements, 787 Bug
> fixes new fixed issues since 2.8.2 .
> 
>  More information about the 2.9.0 release plan can be found here:
> *https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
> *
> 
>  New RC is available at:
> http://home.apache.org/~asuresh/hadoop-2.9.0-RC0/
> 
>  The RC tag in git is: release-2.9.0-RC0, and the latest commit id is:
> 6697f0c18b12f1bdb99cbdf81394091f4fef1f0a
> 
>  The maven artifacts are available via repository.apache.org at:
> *https://repository.apache.org/content/repositories/orgapachehadoop-1065/
> *
> 
>  Please try the release and vote; the vote will run for the usual 5
> days, ending on 11/10/2017 4pm PST time.
> 
> Thanks,
> 
> Arun/Subru

[VOTE] Release Apache Hadoop 2.9.0 (RC0)

[Arun Suresh]

Hi folks,

 Apache Hadoop 2.9.0 is the first stable release of Hadoop 2.9 line and
will be the latest stable/production release for Apache Hadoop - it
includes 30 New Features with 500+ subtasks, 407 Improvements, 787 Bug
fixes new fixed issues since 2.8.2 .

  More information about the 2.9.0 release plan can be found here:
*https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-Version2.9
*

  New RC is available at:
http://home.apache.org/~asuresh/hadoop-2.9.0-RC0/

  The RC tag in git is: release-2.9.0-RC0, and the latest commit id is:
6697f0c18b12f1bdb99cbdf81394091f4fef1f0a

  The maven artifacts are available via repository.apache.org at:
*https://repository.apache.org/content/repositories/orgapachehadoop-1065/
*

  Please try the release and vote; the vote will run for the usual 5
days, ending on 11/10/2017 4pm PST time.

Thanks,

Arun/Subru

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

[Vinod Kumar Vavilapalli]

>   At a minimum, it should at least be using it’s own maven module for a 
> lot of the bits that generates it’s own maven jars so that we can split this 
> functionality up at build/test time.


I expected this to be the case, but looks like it isn't.

There's lot of value in splitting the HDFS code into smaller modules. 
Definitely newer code like Ozone.

When we did this for YARN, initially there were concerns about module 
proliferation, but looking back, my observations have been that it has done us 
far more good than expected. Starting with the fact that we had clients and 
servers modularized independently, as well as servers from other servers, with 
far cleaner contracts than what we had in Hadoop 1 world.

Thanks
+Vinod

Re: [VOTE] Merge yarn-native-services branch into trunk

[Wangda Tan]

Thanks all for the great works!

+1 (Binding). Tried to use native services to build and run applications
successfully.

- Wangda

On Fri, Nov 3, 2017 at 11:50 AM, Arun Suresh  wrote:

> +1 (binding)
>
> Cheers
> -Arun
>
> On Nov 3, 2017 11:44 AM, "Chandni Singh"  wrote:
>
> > +1
> > Thanks,
> > Chandni
> > 
> > From: Jian He 
> > Sent: Monday, October 30, 2017 1:49 PM
> > To: yarn-...@hadoop.apache.org; Hdfs-dev; Hadoop Common;
> > mapreduce-...@hadoop.apache.org
> > Subject: Re: [VOTE] Merge yarn-native-services branch into trunk
> >
> > Few more things:
> >
> > This is the document for trying a non-docker service on YARN.
> > https://github.com/apache/hadoop/blob/yarn-native-
> > services/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/
> > src/site/markdown/yarn-service/QuickStart.md
> >
> > And the document for a docker based service
> > https://github.com/apache/hadoop/blob/yarn-native-
> > services/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/
> > src/site/markdown/yarn-service/Examples.md
> >
> > And the vote lasts 7 days as usual.
> >
> > Thanks,
> > Jian
> >
> > On Oct 30, 2017, at 1:06 PM, Jian He  > e...@hortonworks.com>> wrote:
> >
> > Hi All,
> >
> > I would like to restart the vote for merging yarn-native-services to
> trunk.
> > Since last vote, we have been working on several issues in documentation,
> > DNS, CLI modifications etc. We believe now the feature is in a much
> better
> > shape.
> >
> > Some back ground:
> > At a high level, the following are the key feautres implemented.
> > - YARN-5079[1]. A native YARN framework (ApplicationMaster) to
> orchestrate
> > existing services to YARN either docker or non-docker based.
> > - YARN-4793[2]. A Rest API service embeded in RM (optional)  for user to
> > deploy a service via a simple JSON spec
> > - YARN-4757[3]. Extending today's service registry with a simple DNS
> > service to enable users to discover services deployed on YARN via
> standard
> > DNS lookup
> > - YARN-6419[4]. UI support for native-services on the new YARN UI
> > All these new services are optional and are sitting outside of the
> > existing system, and have no impact on existing system if disabled.
> >
> > Special thanks to a team of folks who worked hard towards this: Billie
> > Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith
> Sharma
> > K S, Sunil G, Akhil PB, Eric Yang. This effort could not be possible
> > without their ideas and hard work.
> > Also thanks Allen for some review and verifications.
> >
> > Thanks,
> > Jian
> >
> > [1] https://issues.apache.org/jira/browse/YARN-5079
> > [2] https://issues.apache.org/jira/browse/YARN-4793
> > [3] https://issues.apache.org/jira/browse/YARN-4757
> > [4] https://issues.apache.org/jira/browse/YARN-6419
> >
> >
> >
> >
> > -
> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >
> >
>

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

[Ravi Prakash]

Hi folks!

Thank you for sharing the design docs and the tremendous amount of work
that has gone into Ozone. I'm grateful that atleast someone is trying to
drastically improve HDFS.

*If* there is a meeting to discuss this merge, could I please also be
invited?

Have we ever thought about distributing the Namenode metadata across nodes
dynamically based on load and RPC times (unlike static federation that we
have now)?

Also, I think a major feature that HDFS still lacks (and a lot of our users
ask for) is BCP / Disaster Recovery. I only bring this up to see if the
choice of proposed design would have implications for that later on.

Thanks,
Ravi

On Fri, Nov 3, 2017 at 1:56 PM, sanjay Radia  wrote:

> Konstantine,
>  Thanks for your comments, questions and feedback. I have attached a
> document to the HDFS-7240 jira
>  that explains a design for scaling HDFS and how Ozone paves the way
> towards the full solution.
>
>
> https://issues.apache.org/jira/secure/attachment/
> 12895963/HDFS%20Scalability%20and%20Ozone.pdf
>
>
> sanjay
>
>
>
>
> > On Oct 28, 2017, at 2:00 PM, Konstantin Shvachko 
> wrote:
> >
> > Hey guys,
> >
> > It is an interesting question whether Ozone should be a part of Hadoop.
> > There are two main reasons why I think it should not.
> >
> > 1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
> > sizable community behind, it looks to me like a whole new project.
> > It is essentially a new storage system, with different (than HDFS)
> > architecture, separate S3-like APIs. This is really great - the World
> sure
> > needs more distributed file systems. But it is not clear why Ozone should
> > co-exist with HDFS under the same roof.
> >
> > 2. Ozone is probably just the first step in rebuilding HDFS under a new
> > architecture. With the next steps presumably being HDFS-10419 and
> > HDFS-8.
> > The design doc for the new architecture has never been published. I can
> > only assume based on some presentations and personal communications that
> > the idea is to use Ozone as a block storage, and re-implement NameNode,
> so
> > that it stores only a partial namesapce in memory, while the bulk of it
> > (cold data) is persisted to a local storage.
> > Such architecture makes me wonder if it solves Hadoop's main problems.
> > There are two main limitations in HDFS:
> >  a. The throughput of Namespace operations. Which is limited by the
> number
> > of RPCs the NameNode can handle
> >  b. The number of objects (files + blocks) the system can maintain. Which
> > is limited by the memory size of the NameNode.
> > The RPC performance (a) is more important for Hadoop scalability than the
> > object count (b). The read RPCs being the main priority.
> > The new architecture targets the object count problem, but in the expense
> > of the RPC throughput. Which seems to be a wrong resolution of the
> tradeoff.
> > Also based on the use patterns on our large clusters we read up to 90% of
> > the data we write, so cold data is a small fraction and most of it must
> be
> > cached.
> >
> > To summarize:
> > - Ozone is a big enough system to deserve its own project.
> > - The architecture that Ozone leads to does not seem to solve the
> intrinsic
> > problems of current HDFS.
> >
> > I will post my opinion in the Ozone jira. Should be more convenient to
> > discuss it there for further reference.
> >
> > Thanks,
> > --Konstantin
> >
> >
> >
> > On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei 
> wrote:
> >
> >> Hello everyone,
> >>
> >>
> >> I would like to start this thread to discuss merging Ozone (HDFS-7240)
> to
> >> trunk. This feature implements an object store which can co-exist with
> >> HDFS. Ozone is disabled by default. We have tested Ozone with cluster
> sizes
> >> varying from 1 to 100 data nodes.
> >>
> >>
> >>
> >> The merge payload includes the following:
> >>
> >>  1.  All services, management scripts
> >>  2.  Object store APIs, exposed via both REST and RPC
> >>  3.  Master service UIs, command line interfaces
> >>  4.  Pluggable pipeline Integration
> >>  5.  Ozone File System (Hadoop compatible file system implementation,
> >> passes all FileSystem contract tests)
> >>  6.  Corona - a load generator for Ozone.
> >>  7.  Essential documentation added to Hadoop site.
> >>  8.  Version specific Ozone Documentation, accessible via service UI.
> >>  9.  Docker support for ozone, which enables faster development cycles.
> >>
> >>
> >> To build Ozone and run ozone using docker, please follow instructions in
> >> this wiki page. https://cwiki.apache.org/confl
> >> uence/display/HADOOP/Dev+cluster+with+docker.
> >>
> >>
> >> We have built a passionate and diverse community to drive this feature
> >> development. As a team, we have achieved significant progress in past 3
> >> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> >> have resolved almost 400 JIRAs by 20+ 

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

[sanjay Radia]

Konstantine, 
 Thanks for your comments, questions and feedback. I have attached a document 
to the HDFS-7240 jira 
 that explains a design for scaling HDFS and how Ozone paves the way towards 
the full solution.


https://issues.apache.org/jira/secure/attachment/12895963/HDFS%20Scalability%20and%20Ozone.pdf


sanjay




> On Oct 28, 2017, at 2:00 PM, Konstantin Shvachko  wrote:
> 
> Hey guys,
> 
> It is an interesting question whether Ozone should be a part of Hadoop.
> There are two main reasons why I think it should not.
> 
> 1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
> sizable community behind, it looks to me like a whole new project.
> It is essentially a new storage system, with different (than HDFS)
> architecture, separate S3-like APIs. This is really great - the World sure
> needs more distributed file systems. But it is not clear why Ozone should
> co-exist with HDFS under the same roof.
> 
> 2. Ozone is probably just the first step in rebuilding HDFS under a new
> architecture. With the next steps presumably being HDFS-10419 and
> HDFS-8.
> The design doc for the new architecture has never been published. I can
> only assume based on some presentations and personal communications that
> the idea is to use Ozone as a block storage, and re-implement NameNode, so
> that it stores only a partial namesapce in memory, while the bulk of it
> (cold data) is persisted to a local storage.
> Such architecture makes me wonder if it solves Hadoop's main problems.
> There are two main limitations in HDFS:
>  a. The throughput of Namespace operations. Which is limited by the number
> of RPCs the NameNode can handle
>  b. The number of objects (files + blocks) the system can maintain. Which
> is limited by the memory size of the NameNode.
> The RPC performance (a) is more important for Hadoop scalability than the
> object count (b). The read RPCs being the main priority.
> The new architecture targets the object count problem, but in the expense
> of the RPC throughput. Which seems to be a wrong resolution of the tradeoff.
> Also based on the use patterns on our large clusters we read up to 90% of
> the data we write, so cold data is a small fraction and most of it must be
> cached.
> 
> To summarize:
> - Ozone is a big enough system to deserve its own project.
> - The architecture that Ozone leads to does not seem to solve the intrinsic
> problems of current HDFS.
> 
> I will post my opinion in the Ozone jira. Should be more convenient to
> discuss it there for further reference.
> 
> Thanks,
> --Konstantin
> 
> 
> 
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei  wrote:
> 
>> Hello everyone,
>> 
>> 
>> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
>> trunk. This feature implements an object store which can co-exist with
>> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
>> varying from 1 to 100 data nodes.
>> 
>> 
>> 
>> The merge payload includes the following:
>> 
>>  1.  All services, management scripts
>>  2.  Object store APIs, exposed via both REST and RPC
>>  3.  Master service UIs, command line interfaces
>>  4.  Pluggable pipeline Integration
>>  5.  Ozone File System (Hadoop compatible file system implementation,
>> passes all FileSystem contract tests)
>>  6.  Corona - a load generator for Ozone.
>>  7.  Essential documentation added to Hadoop site.
>>  8.  Version specific Ozone Documentation, accessible via service UI.
>>  9.  Docker support for ozone, which enables faster development cycles.
>> 
>> 
>> To build Ozone and run ozone using docker, please follow instructions in
>> this wiki page. https://cwiki.apache.org/confl
>> uence/display/HADOOP/Dev+cluster+with+docker.
>> 
>> 
>> We have built a passionate and diverse community to drive this feature
>> development. As a team, we have achieved significant progress in past 3
>> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
>> have resolved almost 400 JIRAs by 20+ contributors/committers from
>> different countries and affiliations. We also want to thank the large
>> number of community members who were supportive of our efforts and
>> contributed ideas and participated in the design of ozone.
>> 
>> 
>> Please share your thoughts, thanks!
>> 
>> 
>> -- Weiwei Yang
>> 
> 
> 
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei  wrote:
> 
>> Hello everyone,
>> 
>> 
>> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
>> trunk. This feature implements an object store which can co-exist with
>> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
>> varying from 1 to 100 data nodes.
>> 
>> 
>> 
>> The merge payload includes the following:
>> 
>>  1.  All services, management scripts
>>  2.  Object store APIs, exposed via both REST and RPC
>>  3.  Master service UIs, command line interfaces
>>  4.  Pluggable pipeline Integration

[jira] [Created] (HDFS-12774) Ozone: OzoneClient: Moving OzoneException from hadoop-hdfs to hadoop-hdfs-client

[Nanda kumar (JIRA)]

Nanda kumar created HDFS-12774:
--

 Summary: Ozone: OzoneClient: Moving OzoneException from 
hadoop-hdfs to hadoop-hdfs-client
 Key: HDFS-12774
 URL: https://issues.apache.org/jira/browse/HDFS-12774
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone
Reporter: Nanda kumar
Assignee: Nanda kumar
Priority: Major


{{OzoneException}} has to be used in hadoop-hdfs-client, since we cannot refer 
classes in hadoop-hdfs from hadoop-hdfs-client moving it to client module.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

[Allen Wittenauer]

> On Nov 3, 2017, at 12:08 PM, Stack  wrote:
> 
> On Sat, Oct 28, 2017 at 2:00 PM, Konstantin Shvachko 
> wrote:
> 
>> It is an interesting question whether Ozone should be a part of Hadoop.
> 
> I don't see a direct answer to this question. Is there one? Pardon me if
> I've not seen it but I'm interested in the response.

+1

Given:

* a completely different set of config files (ozone-site.xml, etc)
* package name is org.apache.hadoop.ozone, not 
org.apache.hadoop.hdfs.ozone

… it doesn’t really seem to want to be part of HDFS, much less Hadoop.

Plus hadoop-hdfs-project/hadoop-hdfs is already a battle zone when it comes to 
unit tests, dependencies, etc [*]

At a minimum, it should at least be using it’s own maven module for a 
lot of the bits that generates it’s own maven jars so that we can split this 
functionality up at build/test time.

At a higher level, this feels a lot like the design decisions that were 
made around yarn-native-services.  This feature is either part of HDFS or it’s 
not. Pick one.  Doing both is incredibly confusing for everyone outside of the 
branch.
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: 答复: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

[Xiaoyu Yao]

Hi Lei,

Thank you for your interest in Ozone. Let me answer each of the
specific questions.

> As the current state of Ozone implementation, what are the major
> benefits of using today’s Ozone over HDFS?


Scale -  HDFS tops out at 500/700 million keys; Ozone primary 
use case is to go beyond that limit. Ozone can scale block space 
and namespace independently. Ozone also provides object store 
semantics which is simpler and scales better


Enabling new workloads on HDFS -  For example, we see lots of 
customers moving to a cloud-like model, where they have a compute 
cluster and storage cluster. This allows them to move workloads to 
the cloud and back seamlessly. We see a marked increase of 
Docker/Kubernetes enabled workloads. The pain point for most 
Docker/VM deployments is lack of storage. Ozone is an excellent 
fit for Docker/Kubernetes deployments.

Ease of Management and use - Ozone has learned very 
valuable lessons from HDFS. It comes with a good set of 
management tools and tries to avoid very complicated setup.


> Giving that its missing features like HDFS-12680 and HDFS-12697, and
> the closing of Hadoop 3.0 release, should we wait for a late merge
> when Ozone is more mature ?

Both HDFS-12680 (lease manager) and HDFS-12697 (Ozone services 
stay disabled in secure setup) are resolved in the past weeks. 
We are targeting the merge for trunk not 3.0.

> Or more generally, why should this merge to a release branch happen
> now, when Ozone is not yet usable by users? Staying on a feature
> branch seems like it's still the right place to Me.

Let me repeat that we are not merging to the 3.0 branch. We are
merging to trunk and we do not intend to backport this to 3.0 code
base.

Ozone is certainly usable. We have written and read billions of keys
into ozone. I would think that it more like Erasure coding when we
merged. We want ozone to be used/tested when people start using 3.1
release. Yes, it is an Alpha feature, having an Alpha release out in
the community is the best way to mature Ozone.


> For the existing HDFS user, could you address the semantic gaps
> between Ozone / Ozone File System and HDFS.

Ozone file system offers a Hadoop compatible file system. For the
first release, we are targeting YARN, Hive, and Spark as the principle
workloads. These applications are functional with Ozone file system.

> It would be great to illustrate what is the expected use cases for
> Ozone giving its different architecture and design decisions?

We expect almost all real use case of ozone to come via Ozone File
System. Hence our focus has been to make sure that (YARN, Hive and
Spark) work well with this system. Ozone file system does the right
magic on behalf of the users for now.

> Like no append, no atomic rename and etc.

This is similar to S3 -- the rise of cloud-based object stores has made 
it very easy for ozone. In fact, the work done by other stacks (Hive, Spark 
etc.) 
to enable big data workload in cloud is extremely helpful for ozone.


> A follow question, was it able to run any of today’s Hadoop
> applications (MR, Spark, Impala, Presto and etc) on Ozone directly, or
> against OZoneFileSystem? I think a performance / scalability gain or
> extended functionality should be the prerequisites for the merge.
> Additionally, I believe such tests will reveal the potential caveats
> if any.

We have run, Mapreduce (pretty much all standard applications along
with Distcp works well), YARN, Hive and Spark against ozone with NO
Modifications to MR,YARN, Hive or Spark.

We have never tried out Impala or Presto, but if they are known to work
well against Hadoop compatible file systems, I am hopeful that they
will work as well. Please feel free to test and report if you run into
any issues.


> * Ozone’s architecture shows great potential to address NN
> scalability.  However it looks like a XXL effort to me, considering
> the fact that 1) the community had multiple unfinished attempts to
> simply separate namespace and block management within the same NN
> process,

You are absolutely correct. We have learned from those experiences. We
think that separating namespace and block space in the same NN process
does not address the core issue of NN scale. And, also as you clearly 
mentioned they are unfinished. 

With Ozone, we have separated out a block service. Once it is known 
to be stable, we will use that in Namenode, thus achieving the full 
separation. Ozone FS and Ozone object store are intermediate steps
 to solving the scale issue for HDFS.

> *and 2) many existing features like snapshot, append, erasure coding,
> and etc, are not straightforward to be implemented in today’s ozone
> design. Could you share your opinions on this matter?

Ozone is well prepared to implement each of these features. We have
many design documents for ozone posted in the sub-JIRAs. For example,
please take a look at the versioning doc to understand how Ozone’s block
layer really offers.

Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

[Stack]

On Sat, Oct 28, 2017 at 2:00 PM, Konstantin Shvachko 
wrote:

> Hey guys,
>
> It is an interesting question whether Ozone should be a part of Hadoop.
>


I don't see a direct answer to this question. Is there one? Pardon me if
I've not seen it but I'm interested in the response.

I ask because IMO the "Hadoop" project is over-stuffed already. Just see
the length of the cc list on this email. Ozone could be standalone. It is a
coherent enough effort.

Thanks,
St.Ack





> There are two main reasons why I think it should not.
>
1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
> sizable community behind, it looks to me like a whole new project.
> It is essentially a new storage system, with different (than HDFS)
> architecture, separate S3-like APIs. This is really great - the World sure
> needs more distributed file systems. But it is not clear why Ozone should
> co-exist with HDFS under the same roof.
>
> 2. Ozone is probably just the first step in rebuilding HDFS under a new
> architecture. With the next steps presumably being HDFS-10419 and
> HDFS-8.
> The design doc for the new architecture has never been published. I can
> only assume based on some presentations and personal communications that
> the idea is to use Ozone as a block storage, and re-implement NameNode, so
> that it stores only a partial namesapce in memory, while the bulk of it
> (cold data) is persisted to a local storage.
> Such architecture makes me wonder if it solves Hadoop's main problems.
> There are two main limitations in HDFS:
>   a. The throughput of Namespace operations. Which is limited by the number
> of RPCs the NameNode can handle
>   b. The number of objects (files + blocks) the system can maintain. Which
> is limited by the memory size of the NameNode.
> The RPC performance (a) is more important for Hadoop scalability than the
> object count (b). The read RPCs being the main priority.
> The new architecture targets the object count problem, but in the expense
> of the RPC throughput. Which seems to be a wrong resolution of the
> tradeoff.
> Also based on the use patterns on our large clusters we read up to 90% of
> the data we write, so cold data is a small fraction and most of it must be
> cached.
>
> To summarize:
> - Ozone is a big enough system to deserve its own project.
> - The architecture that Ozone leads to does not seem to solve the intrinsic
> problems of current HDFS.
>
> I will post my opinion in the Ozone jira. Should be more convenient to
> discuss it there for further reference.
>
> Thanks,
> --Konstantin
>
>
>
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei 
> wrote:
>
> > Hello everyone,
> >
> >
> > I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> > trunk. This feature implements an object store which can co-exist with
> > HDFS. Ozone is disabled by default. We have tested Ozone with cluster
> sizes
> > varying from 1 to 100 data nodes.
> >
> >
> >
> > The merge payload includes the following:
> >
> >   1.  All services, management scripts
> >   2.  Object store APIs, exposed via both REST and RPC
> >   3.  Master service UIs, command line interfaces
> >   4.  Pluggable pipeline Integration
> >   5.  Ozone File System (Hadoop compatible file system implementation,
> > passes all FileSystem contract tests)
> >   6.  Corona - a load generator for Ozone.
> >   7.  Essential documentation added to Hadoop site.
> >   8.  Version specific Ozone Documentation, accessible via service UI.
> >   9.  Docker support for ozone, which enables faster development cycles.
> >
> >
> > To build Ozone and run ozone using docker, please follow instructions in
> > this wiki page. https://cwiki.apache.org/confl
> > uence/display/HADOOP/Dev+cluster+with+docker.
> >
> >
> > We have built a passionate and diverse community to drive this feature
> > development. As a team, we have achieved significant progress in past 3
> > years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> > have resolved almost 400 JIRAs by 20+ contributors/committers from
> > different countries and affiliations. We also want to thank the large
> > number of community members who were supportive of our efforts and
> > contributed ideas and participated in the design of ozone.
> >
> >
> > Please share your thoughts, thanks!
> >
> >
> > -- Weiwei Yang
> >
>
>
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei 
> wrote:
>
> > Hello everyone,
> >
> >
> > I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> > trunk. This feature implements an object store which can co-exist with
> > HDFS. Ozone is disabled by default. We have tested Ozone with cluster
> sizes
> > varying from 1 to 100 data nodes.
> >
> >
> >
> > The merge payload includes the following:
> >
> >   1.  All services, management scripts
> >   2.  Object store APIs, exposed via both REST and RPC
> >   3.  Master service UIs, command line 

Re: [VOTE] Merge yarn-native-services branch into trunk

[Arun Suresh]

+1 (binding)

Cheers
-Arun

On Nov 3, 2017 11:44 AM, "Chandni Singh"  wrote:

> +1
> Thanks,
> Chandni
> 
> From: Jian He 
> Sent: Monday, October 30, 2017 1:49 PM
> To: yarn-...@hadoop.apache.org; Hdfs-dev; Hadoop Common;
> mapreduce-...@hadoop.apache.org
> Subject: Re: [VOTE] Merge yarn-native-services branch into trunk
>
> Few more things:
>
> This is the document for trying a non-docker service on YARN.
> https://github.com/apache/hadoop/blob/yarn-native-
> services/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/
> src/site/markdown/yarn-service/QuickStart.md
>
> And the document for a docker based service
> https://github.com/apache/hadoop/blob/yarn-native-
> services/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/
> src/site/markdown/yarn-service/Examples.md
>
> And the vote lasts 7 days as usual.
>
> Thanks,
> Jian
>
> On Oct 30, 2017, at 1:06 PM, Jian He  e...@hortonworks.com>> wrote:
>
> Hi All,
>
> I would like to restart the vote for merging yarn-native-services to trunk.
> Since last vote, we have been working on several issues in documentation,
> DNS, CLI modifications etc. We believe now the feature is in a much better
> shape.
>
> Some back ground:
> At a high level, the following are the key feautres implemented.
> - YARN-5079[1]. A native YARN framework (ApplicationMaster) to orchestrate
> existing services to YARN either docker or non-docker based.
> - YARN-4793[2]. A Rest API service embeded in RM (optional)  for user to
> deploy a service via a simple JSON spec
> - YARN-4757[3]. Extending today's service registry with a simple DNS
> service to enable users to discover services deployed on YARN via standard
> DNS lookup
> - YARN-6419[4]. UI support for native-services on the new YARN UI
> All these new services are optional and are sitting outside of the
> existing system, and have no impact on existing system if disabled.
>
> Special thanks to a team of folks who worked hard towards this: Billie
> Rinaldi, Gour Saha, Vinod Kumar Vavilapalli, Jonathan Maron, Rohith Sharma
> K S, Sunil G, Akhil PB, Eric Yang. This effort could not be possible
> without their ideas and hard work.
> Also thanks Allen for some review and verifications.
>
> Thanks,
> Jian
>
> [1] https://issues.apache.org/jira/browse/YARN-5079
> [2] https://issues.apache.org/jira/browse/YARN-4793
> [3] https://issues.apache.org/jira/browse/YARN-4757
> [4] https://issues.apache.org/jira/browse/YARN-6419
>
>
>
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

[jira] [Created] (HDFS-12773) RBF: Improve State Store FS implementation

[JIRA]

Íñigo Goiri created HDFS-12773:
--

 Summary: RBF: Improve State Store FS implementation
 Key: HDFS-12773
 URL: https://issues.apache.org/jira/browse/HDFS-12773
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Íñigo Goiri






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12772) RBF: Track Router states

[JIRA]

Íñigo Goiri created HDFS-12772:
--

 Summary: RBF: Track Router states
 Key: HDFS-12772
 URL: https://issues.apache.org/jira/browse/HDFS-12772
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Íñigo Goiri
Priority: Normal






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/

[Nov 3, 2017 12:15:33 AM] (Arun Suresh) HADOOP-15013. Fix ResourceEstimator 
findbugs issues. (asuresh)
[Nov 3, 2017 12:39:23 AM] (subru) YARN-7432. Fix DominantResourceFairnessPolicy 
serializable findbugs
[Nov 3, 2017 1:55:29 AM] (sunilg) YARN-7410. Cleanup FixedValueResource to 
avoid dependency to
[Nov 3, 2017 4:27:35 AM] (xiao) HDFS-12682. ECAdmin -listPolicies will always 
show
[Nov 3, 2017 4:29:53 AM] (inigoiri) YARN-7434. Router getApps REST invocation 
fails with multiple RMs.
[Nov 3, 2017 4:53:13 AM] (xiao) HDFS-12725. 
BlockPlacementPolicyRackFaultTolerant fails with very uneven
[Nov 3, 2017 6:15:50 AM] (sunilg) YARN-7392. Render cluster information on new 
YARN web ui. Contributed by




-1 overall


The following subsystems voted -1:
asflicense unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Unreaped Processes :

   hadoop-hdfs:5 
   hadoop-mapreduce-client-jobclient:1 

Failed junit tests :

   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 
   hadoop.hdfs.TestDecommission 
   hadoop.hdfs.server.mover.TestMover 
   
hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
 
   hadoop.hdfs.server.balancer.TestBalancer 
   hadoop.hdfs.TestReconstructStripedFile 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170 
   hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean 
   hadoop.hdfs.server.balancer.TestBalancerRPCDelay 
   hadoop.hdfs.TestReadStripedFileWithDNFailure 
   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped 
   hadoop.yarn.server.TestDiskFailures 
   hadoop.yarn.client.api.impl.TestAMRMClient 
   hadoop.yarn.sls.TestSLSRunner 

Timed out junit tests :

   
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA 
   
org.apache.hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA 
   
org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA 
   org.apache.hadoop.mapred.pipes.TestPipeApplication 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/diff-compile-javac-root.txt
  [280K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/whitespace-eol.txt
  [8.8M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/whitespace-tabs.txt
  [292K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/diff-javadoc-javadoc-root.txt
  [760K]

   UnreapedProcessesLog:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-reaper.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-reaper.txt
  [4.0K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [1.2M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [64K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
  [84K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/579/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt
  [8.0K]

   

[jira] [Created] (HDFS-12771) Add genstamp and block size to metasave Corrupt blocks list

[Kuhu Shukla (JIRA)]

Kuhu Shukla created HDFS-12771:
--

 Summary: Add genstamp and block size to metasave Corrupt blocks 
list
 Key: HDFS-12771
 URL: https://issues.apache.org/jira/browse/HDFS-12771
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Kuhu Shukla
Assignee: Kuhu Shukla
Priority: Minor


For corrupt blocks in metasave, adding genstamp and blocksize can be useful 
instead of just the blockIds.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-3821) Backport HDFS-3626 to branch-1 (Creating file with invalid path can corrupt edit log)

[Andras Bokor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Bokor resolved HDFS-3821.

Resolution: Won't Fix

> Backport HDFS-3626 to branch-1 (Creating file with invalid path can corrupt 
> edit log)
> -
>
> Key: HDFS-3821
> URL: https://issues.apache.org/jira/browse/HDFS-3821
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 1.0.0
>Reporter: Eli Collins
>Priority: Major
>
> Per [Todd's 
> comment|https://issues.apache.org/jira/browse/HDFS-3626?focusedCommentId=13413509=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13413509]
>  this issue affects v1 as well though the problem isn't as obvious because 
> the shell doesn't use the Path(URI) constructor. To test the server side Todd 
> modified the touchz command to use new Path(new URI(src)) and was able to 
> reproduce the issue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12770) Add doc about how to disable client socket cache

[Weiwei Yang (JIRA)]

Weiwei Yang created HDFS-12770:
--

 Summary: Add doc about how to disable client socket cache
 Key: HDFS-12770
 URL: https://issues.apache.org/jira/browse/HDFS-12770
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client
Reporter: Weiwei Yang
Assignee: Weiwei Yang
Priority: Minor


After HDFS-3365, client socket cache (PeerCache) can be disabled, but there is 
no doc about this. We should add some doc in hdfs-default.xml to instruct user 
how to disable it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Cutting branch-2.9

[Arun Suresh]

Quick update folks.

We just cut branch-2.9.0, and set the mvn version to 2.9.0
We've also updated the version on branch-2.9 to 2.9.1-SNAPSHOT.

Expect an RC in the next 24 hrs :)

Cheers
-Arun/Subru

On Tue, Oct 31, 2017 at 5:24 PM, Arun Suresh  wrote:

> Hi Junping,
>
> > In the mean time, branch-2.9 should be reserved for next 2.9 point
> release (2.9.1) and branch-2 should be reserved for next minor release
> (2.10.0 or whatever name it is). Thoughts?
> Yup, That is the current plan. We've updated mvn versions in branch-2 to
> point to 2.10.0-SNAPSHOT.
> Once we cut branch-2.9.0, we will update mvn versions on branch-2.9.0 to
> 2.9.0 and set versions on branch-2.9 to 2.9.1-SNAPSHOT (We plan to do that
> by Thursday)
>
> Cheers
> -Arun
>
> On Tue, Oct 31, 2017 at 2:35 PM, 俊平堵  wrote:
>
>> Hi Arun/Subru,
>> Thanks for updating on 2.9.0 release progress. A quick question here:
>> are we planning to release from branch-2.9 directly?
>> I doubt this as it seems against our current branch release practice (
>> https://wiki.apache.org/hadoop/HowToRelease#Branching). To get rid of
>> any confusion, I would suggest to cut-off branch-2.9.0 for 2.9.0 release
>> work. In the mean time, branch-2.9 should be reserved for next 2.9 point
>> release (2.9.1) and branch-2 should be reserved for next minor release
>> (2.10.0 or whatever name it is). Thoughts?
>>
>> bq. @Junping, lets move the jdiff conversation to separate thread.
>> Sure. I will reply jdiff in separated thread.
>>
>> Thanks,
>>
>> Junping
>>
>> 2017-10-31 13:44 GMT-07:00 Arun Suresh :
>>
>>> Hello folks
>>>
>>> We just cut branch-2.9 since all the critical/blocker issues are now
>>> resolved.
>>> We plan to perform some sanity checks for the rest of the week and cut
>>> branch-2.9.0 and push out an RC0 by the end of the week.
>>>
>>> Kindly refrain from committing to branch-2.9 without giving us a heads
>>> up.
>>>
>>> @Junping, lets move the jdiff conversation to separate thread.
>>>
>>> Cheers
>>> -Arun/Subru
>>>
>>> On Mon, Oct 30, 2017 at 12:39 PM, Subru Krishnan 
>>> wrote:
>>>
>>> > We want to give heads up that we are going to cut branch-2.9 tomorrow
>>> > morning.
>>> >
>>> > We are down to 3 blockers and they all are close to being committed
>>> > (thanks everyone):
>>> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.9+Release
>>> >
>>> > There are 4 other non-blocker JIRAs that are targeted for 2.9.0 which
>>> are
>>> > close to completion.
>>> >
>>> > Folks who are working/reviewing these, kindly prioritize accordingly so
>>> > that we can make the release on time.
>>> > https://issues.apache.org/jira/browse/YARN-7398?filter=12342468
>>> >
>>> > Thanks in advance!
>>> >
>>> > -Subru/Arun
>>> >
>>> >
>>> >
>>>
>>
>>
>

Re: 答复: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

[Lei Xu]

Hey,  Weiwei and Jitendra

Thanks a lot for this large effort to bring us ozone.

* As the current state of Ozone implementation, what are the major
benefits of using today’s Ozone over HDFS?  Giving that its missing
features like HDFS-12680 and HDFS-12697, being disabled by default,
and the closing of Hadoop 3.0 release, should we wait for a late merge
when Ozone is more mature ? Or more generally, why should this merge
to a release branch happen now, when Ozone is not yet usable by users?
Staying on a feature branch seems like it's still the right place to
me.
* For the existing HDFS user, could you address the semantic gaps
between Ozone / Ozone File System and HDFS. It would be great to
illustrate what is the expected use cases for Ozone giving its
different architecture and design decisions?  Like no append, no
atomic rename and etc.
* A follow question, was it able to run any of today’s Hadoop
applications (MR, Spark, Impala, Presto and etc) on Ozone directly, or
against OZoneFileSystem? I think a performance / scalability gain or
extended functionality should be the prerequisites for the merge.
Additionally, I believe such tests will reveal the potential caveats
if any.
* Ozone’s architecture shows great potential to address NN
scalability.  However it looks like a XXL effort to me, considering
the fact that 1) the community had multiple unfinished attempts to
simply separate namespace and block management within the same NN
process, and 2) many existing features like snapshot, append, erasure
coding, and etc, are not straightforward to be implemented in today’s
ozone design. Could you share your opinions on this matter?
* How stable is the ozone client? Should we mark them as unstable for
now? Also giving the significant difference between OzoneClient and
HdfsClient, should move it to a separated package or even a project? I
second Konstantin’s option to separate ozone from HDFS.
* Please add sections to the end-user and system admin oriented
documents for deploying and operating SCM, KSM, and also the chunk
servers on DataNodes. Additionally, the introduction in
“OZoneGettingStarted.md” is still building ozone from feature branch
HDFS-7240.

Best regards,

On Mon, Oct 23, 2017 at 11:10 AM, Jitendra Pandey
 wrote:
> I have filed https://issues.apache.org/jira/browse/HDFS-12697 to ensure ozone 
> stays disabled in a secure environment.
> Since ozone is disabled by default and will not come with security on, it 
> will not expose any new attack surface in a Hadoop deployment.
> Ozone security effort will need a detailed design and discussion on a 
> community jira. Hopefully, that effort will start soon after the merge.
>
> Thanks
> jitendra
>
> On 10/20/17, 2:40 PM, "larry mccay"  wrote:
>
> All -
>
> I broke this list of questions out into a separate DISCUSS thread where we
> can iterate over how a security audit process at merge time might look and
> whether it is even something that we want to take on.
>
> I will try and continue discussion on that thread and drive that to some
> conclusion before bringing it into any particular merge discussion.
>
> thanks,
>
> --larry
>
> On Fri, Oct 20, 2017 at 12:37 PM, larry mccay  wrote:
>
> > I previously sent this same email from my work email and it doesn't seem
> > to have gone through - resending from apache account (apologizing up 
> from
> > for the length)
> >
> > For such sizable merges in Hadoop, I would like to start doing security
> > audits in order to have an initial idea of the attack surface, the
> > protections available for known threats, what sort of configuration is
> > being used to launch processes, etc.
> >
> > I dug into the architecture documents while in the middle of this list -
> > nice docs!
> > I do intend to try and make a generic check list like this for such
> > security audits in the future so a lot of this is from that but I tried 
> to
> > also direct specific questions from those docs as well.
> >
> > 1. UIs
> > I see there are at least two UIs - Storage Container Manager and Key 
> Space
> > Manager. There are a number of typical vulnerabilities that we find in 
> UIs
> >
> > 1.1. What sort of validation is being done on any accepted user input?
> > (pointers to code would be appreciated)
> > 1.2. What explicit protections have been built in for (pointers to code
> > would be appreciated):
> >   1.2.1. cross site scripting
> >   1.2.2. cross site request forgery
> >   1.2.3. click jacking (X-Frame-Options)
> > 1.3. What sort of authentication is required for access to the UIs?
> > 1.4. What authorization is available for determining who can access what
> > capabilities of the UIs for either viewing, modifying data or affecting
> > object stores and related processes?
> > 1.5. Are the UIs built