Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk

[Sangjin Lee]

Thanks Rohith for that update! Sounds good. Let us know when that testing
is complete too.

On Sat, Aug 19, 2017 at 3:34 AM, Rohith Sharma K S <
rohithsharm...@apache.org> wrote:

> Hi Sangjin
>
> Thanks for bringing this point.
> We did similar exercise with current YARN-5355 branch today. Tests are
> validated against default configuration and timeline server v.1.5 as well.
>
> All YARN major features such as RM HA/Restart/work-preserving-restart, NM
> restart, scheduling, sample MR/distributes shell jobs run are validated.
> Everything looks fine.
>
> This testing will also be done once YARN-5355 branch code freezed
> completely as well in couple of days.
>
> Thanks & Regards
> Rohith Sharma K S
>
> On 18 August 2017 at 22:56, Sangjin Lee  wrote:
>
> > Kudos to Vrushali and the team for getting ready for this large and
> > important feature! I know a huge team effort went into this. I look
> forward
> > to seeing this merged.
> >
> > I'd like to ask one piece of due diligence. Could you please inspect
> > rigorously to ensure that when disabled Timeline Service v.2 does not
> > impact other features in any way? We did a similar exercise when we had
> the
> > first drop, and it would be good to repeat that... Thanks!
> >
> > Sangjin
> >
> > On Wed, Aug 16, 2017 at 11:44 AM, Andrew Wang 
> > wrote:
> >
> > > Great, thanks Vrushali! Sounds good to me.
> > >
> > > I have a few procedural release notes comments I'll put on YARN-5355,
> to
> > > make sure we advertise this to our users appropriately.
> > >
> > > On Wed, Aug 16, 2017 at 11:32 AM, Vrushali Channapattan <
> > > vrushal...@gmail.com> wrote:
> > >
> > > > Hi Andrew,
> > > >
> > > > Thanks for your response!
> > > >
> > > > There have been no changes to existing APIs since alpha1.
> > > >
> > > > We at Twitter have tested the feature to demonstrate it works at what
> > we
> > > > consider moderate scale but this did not include the security related
> > > > testing. The security testing is in progress at present by Timeline
> > > Service
> > > > V2 team in the community and we think we will have more details on
> this
> > > > very soon.
> > > >
> > > > About the jiras under YARN-5355: Only 3 of those sub-tasks are what
> we
> > > > think of as "merge-blockers". The issues being targeted for merge are
> > in
> > > > [link1] below. There are about 59 jiras of which 56 are completed.
> > > >
> > > > We plan to make a new umbrella jira after the merge to trunk. We will
> > > then
> > > > create a new branch with the new jira name and move these open jiras
> > > under
> > > > YARN-5355 as subtasks of that new umbrella jira.
> > > >
> > > > thanks
> > > > Vrushali
> > > > [link1] https://issues.apache.org/jira/projects/YARN/versions/
> 12337991
> > > >
> > > >
> > > > On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang <
> > andrew.w...@cloudera.com>
> > > > wrote:
> > > >
> > > >> Hi Vrushali,
> > > >>
> > > >> Glad to hear this major dev milestone is nearing completion!
> > > >>
> > > >> Repeating my request on other merge [DISCUSS] threads, could you
> > comment
> > > >> on testing and API stability of this merge? Our timeline for beta1
> is
> > > about
> > > >> a month out, so there's not much time to fix things beforehand.
> > > >>
> > > >> Looking at YARN-5355 there are also many unresolved subtasks. Should
> > > most
> > > >> of these be moved out to a new umbrella? I'm wondering what needs to
> > be
> > > >> completed before sending the merge vote.
> > > >>
> > > >> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex
> > the
> > > >> beta1 release date for this feature than others. Hopefully that
> won't
> > be
> > > >> necessary though :)
> > > >>
> > > >> Best,
> > > >> Andrew
> > > >>
> > > >> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
> > > >> vrushalic2...@gmail.com> wrote:
> > > >>
> > > >>> Looks like some of the hyperlinks appear messed up, my apologies,
> > > >>> resending
> > > >>> the same email with hopefully better looking content:
> > > >>>
> > > >>> Hi All,
> > > >>>
> > > >>> I'd like to open a discussion for merging Timeline Service v2
> > > (YARN-5355)
> > > >>> to trunk in a few weeks.
> > > >>>
> > > >>> We have previously completed one merge onto trunk [1] and Timeline
> > > >>> Service
> > > >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> > > >>>
> > > >>> Since then, we have been working on extending the capabilities of
> > > >>> Timeline
> > > >>> Service v2 in a feature branch [2].  There are a few related issues
> > > >>> pending
> > > >>> that are being actively worked upon and tested. As soon as they are
> > > >>> resolved, we plan on starting a merge vote within the next two
> weeks.
> > > The
> > > >>> goal is to get this into hadoop3 beta.
> > > >>>
> > > >>> We have paid close attention to ensure that  once disabled Timeline
> > > >>> Service
> > > >>> v2 does not impact existing functionality when disabled (by
> default).
> > > >>>
> > > 

Re: Branch merges and 3.0.0-beta1 scope

[Wangda Tan]

Andrew,

Thanks for your help to pushing this release.

Echoing what Vinod said, all contributors in these branches are putting
months to years of time working on these features, we don't have to decide
excluded features now since we have 25 days till 3.0-beta1 planned release
time.

The best approach to stabilize feature is to let people try that, instead
of waiting for feature becomes perfect. For features which can be turned
off, I think we should consider to bring it in if it is end-to-end ready. I
will try best to help merge efforts of YARN-3926 branch to trunk before Sep
15, and I'm OK with moving to the next release train if we fail to merge
the feature before release date.

Thanks,
Wangda


On Mon, Aug 21, 2017 at 2:22 PM, Vinod Kumar Vavilapalli  wrote:

> Steve,
>
> You can be strict & ruthless about the timelines. Anything that doesn’t
> get in by mid-September, as was originally planned, can move to the next
> release - whether it is feature work on branches or feature work on trunk.
>
> The problem I see here is that code & branches being worked on for a year
> are now (apparently) close to being done and we are telling them to hold
> for 7 more months - this is not a reasonable ask..
>
> If you are advocating for a 3.1 plan, I’m sure one of these branch
> ‘owners’ can volunteer. But this is how you get competing releases and
> split bandwidth.
>
> As for compatibility / testing etc, it seems like there is a belief that
> the current ‘scoped’ features are all tested well in these areas and so
> adding more is going to hurt the release. There is no way this is the
> reality, trunk has so many features that have been landing for years, the
> only way we can collectively attempt towards making this stable is by
> getting as many parties together as possible, each verifying stuff that
> they need. Not by excluding specific features.
>
> +Vinod
>
> > This is one of those curse-of-cadence things: The higher your release
> cadence, the less pressure to get "everything in". With a slower cadence,
> more pressure to get stuff in, more pressure to hold up the release, slows
> the cadence, gets even more stuff in, etc. etc.
> >
> > - Andrew has been working on the release for months, we all need to
> appreciate how much hard work that is and has been, especially for what is
> going to be a major release.
> >
> > - We know that things will be unstable in 3.0; Andrew's concern is about
> making sure that the newest, unstablest (?) features can at least be
> bypassed if there are problems. I we should also call out in the release
> notes what we think are the unstable bits where people need to use caution
> (example: S3Guard in "authoritative" mode)
> >
> > - Anything related to wire compatibility has been problematic in the
> past; I think it's essential that whatever packets get sent around are
> going to be stable, so changes there need to be in, or at least the
> payloads set up ready for the features. Same for new public APIs.
> >
> > - As fpr the rest, I don't know. I think being strict about it and
> ruthless in assessing the feature's stability & consequences of postponing
> the feature until a Hadoop 3.1 release in Jan/Feb, with a plan to ship then
> and follow up with a 3.2 in the summer.
> >
> > Then: start planning that 3.1 release. Maybe I should put my hand up as
> release manager for that one. Then everyone would realise how amenable
> Andrew is being today.
> >
> >
> > One other thing: alongside the big branches, there's the eternal backlog
> of small patches. We should organise spending a few days updating,
> reviewing & merging them in
> >
> > -Steve
> >
> >
> > -
> > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org  hdfs-dev-unsubscr...@hadoop.apache.org>
> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> 
>

Re: Branch merges and 3.0.0-beta1 scope

[Vinod Kumar Vavilapalli]

Steve,

You can be strict & ruthless about the timelines. Anything that doesn’t get in 
by mid-September, as was originally planned, can move to the next release - 
whether it is feature work on branches or feature work on trunk.

The problem I see here is that code & branches being worked on for a year are 
now (apparently) close to being done and we are telling them to hold for 7 more 
months - this is not a reasonable ask..

If you are advocating for a 3.1 plan, I’m sure one of these branch ‘owners’ can 
volunteer. But this is how you get competing releases and split bandwidth.

As for compatibility / testing etc, it seems like there is a belief that the 
current ‘scoped’ features are all tested well in these areas and so adding more 
is going to hurt the release. There is no way this is the reality, trunk has so 
many features that have been landing for years, the only way we can 
collectively attempt towards making this stable is by getting as many parties 
together as possible, each verifying stuff that they need. Not by excluding 
specific features.

+Vinod

> This is one of those curse-of-cadence things: The higher your release 
> cadence, the less pressure to get "everything in". With a slower cadence, 
> more pressure to get stuff in, more pressure to hold up the release, slows 
> the cadence, gets even more stuff in, etc. etc.
> 
> - Andrew has been working on the release for months, we all need to 
> appreciate how much hard work that is and has been, especially for what is 
> going to be a major release.
> 
> - We know that things will be unstable in 3.0; Andrew's concern is about 
> making sure that the newest, unstablest (?) features can at least be bypassed 
> if there are problems. I we should also call out in the release notes what we 
> think are the unstable bits where people need to use caution (example: 
> S3Guard in "authoritative" mode)
> 
> - Anything related to wire compatibility has been problematic in the past; I 
> think it's essential that whatever packets get sent around are going to be 
> stable, so changes there need to be in, or at least the payloads set up ready 
> for the features. Same for new public APIs.
> 
> - As fpr the rest, I don't know. I think being strict about it and ruthless 
> in assessing the feature's stability & consequences of postponing the feature 
> until a Hadoop 3.1 release in Jan/Feb, with a plan to ship then and follow up 
> with a 3.2 in the summer.
> 
> Then: start planning that 3.1 release. Maybe I should put my hand up as 
> release manager for that one. Then everyone would realise how amenable Andrew 
> is being today.
> 
> 
> One other thing: alongside the big branches, there's the eternal backlog of 
> small patches. We should organise spending a few days updating, reviewing & 
> merging them in
> 
> -Steve
> 
> 
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org 
> 
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org 
> 

[jira] [Created] (HADOOP-14799) Update XXX to 4.41.1

[Ray Chiang (JIRA)]

Ray Chiang created HADOOP-14799:
---

 Summary: Update XXX to 4.41.1
 Key: HADOOP-14799
 URL: https://issues.apache.org/jira/browse/HADOOP-14799
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Ray Chiang


Update the dependency

com.nimbusds:nimbus-jose-jwt:3.9

to the latest (4.41.1)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Merge HADOOP-13345 (S3Guard feature branch)

[Lei Xu]

+1 (binding)

S3guard significantly extends the use cases of Hadoop with S3. I was
involved in the early days of the development.  Really appreciate the
high quality work and collaborative environment from the Hadoop
community.

Best,

On Mon, Aug 21, 2017 at 1:09 PM, Aaron Fabbri  wrote:
> + Eddy Xu (having list issues)
>
>
>
> On Mon, Aug 21, 2017 at 8:10 AM, Steve Loughran 
> wrote:
>>
>>
>> On 18 Aug 2017, at 17:51, John Zhuge
>> > wrote:
>>
>> That will be great. Please record it if possible.
>>
>> good idea. I'll do a video & demo, that way I can avoid fielding hard
>> questions
>>
>
> Hah!
>
> -AF
>
>
>>
>>
>> On Fri, Aug 18, 2017 at 4:12 AM, Steve Loughran
>> > wrote:
>>
>> I can do a demo of this next week if people are interested
>>
>> > On 17 Aug 2017, at 23:07, Aaron Fabbri
>> > > wrote:
>> >
>> > Hello,
>> >
>> > I'd like to open a vote (7 days, ending August 24 at 3:10 PST) to merge
>> > the
>> > HADOOP-13345 feature branch into trunk.
>> >
>> > This branch contains the new S3Guard feature which adds metadata
>> > consistency features to the S3A client.  Formatted site documentation
>> > can
>> > be found here:
>> >
>> >
>> > https://github.com/apache/hadoop/blob/HADOOP-13345/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md
>> >
>> > The current patch against trunk is posted here:
>> >
>> > https://issues.apache.org/jira/browse/HADOOP-13998
>> >
>> > The branch modifies the s3a portion of the hadoop-tools/hadoop-aws
>> > module:
>> >
>> > - The feature is off by default, and care has been taken to insure it
>> > has
>> > no impact when disabled.
>> > - S3Guard can be enabled with the production database which is backed by
>> > DynamoDB, or with a local, in-memory implementation that facilitates
>> > integration testing without having to pay for a database.
>> > - getFileStatus() as well as directory listing consistency has been
>> > implemented and thoroughly tested, including delete tracking.
>> > - Convenient Maven profiles for testing with and without S3Guard.
>> > - New failure injection code and integration tests that exercise it.  We
>> > use timers and a wrapper around the Amazon SDK client object to force
>> > consistency delays to occur.  This allows us to assert that S3Guard
>> > works
>> > as advertised.  This will be extended with more types of failure
>> > injection
>> > to continue hardening the S3A client.
>> >
>> > Outside of hadoop-tools/hadoop-aws's s3a directory there are some minor
>> > changes:
>> >
>> > - core-default.xml defaults and documentation for s3guard parameters.
>> > - A couple additional FS contract test cases around rename.
>> > - More goodies in LambdaTestUtils
>> > - A new CLI tool for inspecting and manipulating S3Guard features,
>> > including the backing MetadataStore database.
>> >
>> > This branch has seen extensive testing as well as use in production.
>> > This
>> > branch makes significant improvements to S3A's test toolkit as well.
>> >
>> > Performance is typically on par with, and in some cases better than, the
>> > existing S3A code without S3Guard enabled.
>> >
>> > This feature was developed with contributions and feedback from many
>> > people.  I'd like to thank everyone who worked on HADOOP-13345 as well
>> > as
>> > all of those who contributed feedback and work on the original design
>> > document.
>> >
>> > This is the first major Apache Hadoop project I've worked on from start
>> > to
>> > finish, and I've really enjoyed it.  Please shout if I've missed
>> > anything
>> > important here or in the VOTE process.
>> >
>> > Cheers,
>> > Aaron Fabbri
>>
>>
>> -
>> To unsubscribe, e-mail:
>> common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail:
>> common-dev-h...@hadoop.apache.org
>>
>>
>>
>>
>> --
>> John
>>
>



-- 
Lei (Eddy) Xu
Software Engineer, Cloudera

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Merge HADOOP-13345 (S3Guard feature branch)

[Aaron Fabbri]

+ Eddy Xu (having list issues)



On Mon, Aug 21, 2017 at 8:10 AM, Steve Loughran 
wrote:

>
> On 18 Aug 2017, at 17:51, John Zhuge > wrote:
>
> That will be great. Please record it if possible.
>
> good idea. I'll do a video & demo, that way I can avoid fielding hard
> questions
>
>
Hah!

-AF



>
> On Fri, Aug 18, 2017 at 4:12 AM, Steve Loughran  mailto:ste...@hortonworks.com>> wrote:
>
> I can do a demo of this next week if people are interested
>
> > On 17 Aug 2017, at 23:07, Aaron Fabbri  b...@cloudera.com>> wrote:
> >
> > Hello,
> >
> > I'd like to open a vote (7 days, ending August 24 at 3:10 PST) to merge
> the
> > HADOOP-13345 feature branch into trunk.
> >
> > This branch contains the new S3Guard feature which adds metadata
> > consistency features to the S3A client.  Formatted site documentation can
> > be found here:
> >
> > https://github.com/apache/hadoop/blob/HADOOP-13345/
> hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md
> >
> > The current patch against trunk is posted here:
> >
> > https://issues.apache.org/jira/browse/HADOOP-13998
> >
> > The branch modifies the s3a portion of the hadoop-tools/hadoop-aws
> module:
> >
> > - The feature is off by default, and care has been taken to insure it has
> > no impact when disabled.
> > - S3Guard can be enabled with the production database which is backed by
> > DynamoDB, or with a local, in-memory implementation that facilitates
> > integration testing without having to pay for a database.
> > - getFileStatus() as well as directory listing consistency has been
> > implemented and thoroughly tested, including delete tracking.
> > - Convenient Maven profiles for testing with and without S3Guard.
> > - New failure injection code and integration tests that exercise it.  We
> > use timers and a wrapper around the Amazon SDK client object to force
> > consistency delays to occur.  This allows us to assert that S3Guard works
> > as advertised.  This will be extended with more types of failure
> injection
> > to continue hardening the S3A client.
> >
> > Outside of hadoop-tools/hadoop-aws's s3a directory there are some minor
> > changes:
> >
> > - core-default.xml defaults and documentation for s3guard parameters.
> > - A couple additional FS contract test cases around rename.
> > - More goodies in LambdaTestUtils
> > - A new CLI tool for inspecting and manipulating S3Guard features,
> > including the backing MetadataStore database.
> >
> > This branch has seen extensive testing as well as use in production.
> This
> > branch makes significant improvements to S3A's test toolkit as well.
> >
> > Performance is typically on par with, and in some cases better than, the
> > existing S3A code without S3Guard enabled.
> >
> > This feature was developed with contributions and feedback from many
> > people.  I'd like to thank everyone who worked on HADOOP-13345 as well as
> > all of those who contributed feedback and work on the original design
> > document.
> >
> > This is the first major Apache Hadoop project I've worked on from start
> to
> > finish, and I've really enjoyed it.  Please shout if I've missed anything
> > important here or in the VOTE process.
> >
> > Cheers,
> > Aaron Fabbri
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org common-dev-unsubscr...@hadoop.apache.org>
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org common-dev-h...@hadoop.apache.org>
>
>
>
>
> --
> John
>
>

[jira] [Created] (HADOOP-14798) Update sshd-core and related mina-core library

[Ray Chiang (JIRA)]

Ray Chiang created HADOOP-14798:
---

 Summary: Update sshd-core and related mina-core library
 Key: HADOOP-14798
 URL: https://issues.apache.org/jira/browse/HADOOP-14798
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Ray Chiang


Update the dependencies

org.apache.mina:mina-core:2.0.0-M5
org.apache.sshd:sshd-core:0.14.0

mina-core can be updated to 2.0.16 and sshd-core to 1.6.0




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14797) Update re2j version to 1.1

[Ray Chiang (JIRA)]

Ray Chiang created HADOOP-14797:
---

 Summary: Update re2j version to 1.1
 Key: HADOOP-14797
 URL: https://issues.apache.org/jira/browse/HADOOP-14797
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Ray Chiang


Update the dependency

com.google.re2j:re2j:1.0

to the latest (1.1).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14796) Update json-simple version to 1.1.1

[Ray Chiang (JIRA)]

Ray Chiang created HADOOP-14796:
---

 Summary: Update json-simple version to 1.1.1
 Key: HADOOP-14796
 URL: https://issues.apache.org/jira/browse/HADOOP-14796
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Ray Chiang


Update the dependency

com.googlecode.json-simple:json-simple:1.1

to the latest (1.1.1).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14795) TestMapFileOutputFormat missing @after annotation

[Ajay Kumar (JIRA)]

Ajay Kumar created HADOOP-14795:
---

 Summary: TestMapFileOutputFormat missing @after annotation
 Key: HADOOP-14795
 URL: https://issues.apache.org/jira/browse/HADOOP-14795
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ajay Kumar
Assignee: Ajay Kumar
Priority: Minor


TestMapFileOutputFormat missing @after annotation. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/499/

[Aug 20, 2017 2:54:09 PM] (Arun Suresh) YARN-6979. Add flag to notify all types 
of container updates to NM via
[Aug 20, 2017 5:24:05 PM] (Arun Suresh) YARN-6979. [Addendum patch] Fixed 
classname and added javadocs.
[Aug 21, 2017 5:36:28 AM] (kai.zheng) HADOOP-14194. Aliyun OSS should not use 
empty endpoint as default.




-1 overall


The following subsystems voted -1:
findbugs unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 
   Hard coded reference to an absolute pathname in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
 At DockerLinuxContainerRuntime.java:absolute pathname in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
 At DockerLinuxContainerRuntime.java:[line 490] 

Failed junit tests :

   hadoop.net.TestDNS 
   hadoop.hdfs.server.namenode.TestNameNodeMXBean 
   
hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
 
   hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped 
   hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem 
   
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation 
   hadoop.yarn.sls.appmaster.TestAMSimulator 
   hadoop.yarn.sls.nodemanager.TestNMSimulator 

Timed out junit tests :

   org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean 
   
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/499/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/499/artifact/out/diff-compile-javac-root.txt
  [296K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/499/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/499/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/499/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/499/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/499/artifact/out/whitespace-eol.txt
  [11M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/499/artifact/out/whitespace-tabs.txt
  [1.2M]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/499/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/499/artifact/out/diff-javadoc-javadoc-root.txt
  [1.9M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/499/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [148K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/499/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [236K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/499/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt
  [16K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/499/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [64K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/499/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt
  [16K]

Powered by Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14794) Standalone MiniKdc server

[John Zhuge (JIRA)]

John Zhuge created HADOOP-14794:
---

 Summary: Standalone MiniKdc server
 Key: HADOOP-14794
 URL: https://issues.apache.org/jira/browse/HADOOP-14794
 Project: Hadoop Common
  Issue Type: New Feature
  Components: security, test
Affects Versions: 2.7.0
Reporter: John Zhuge
Assignee: John Zhuge


Add a new subcommand {{hadoop minikdc}} to start a standalone MiniKdc server. 
This will make it easier to test Kerberos in pseudo-distributed mode without an 
external KDC server.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Merge HADOOP-13345 (S3Guard feature branch)

[Steve Loughran]

On 18 Aug 2017, at 17:51, John Zhuge 
> wrote:

That will be great. Please record it if possible.

good idea. I'll do a video & demo, that way I can avoid fielding hard questions


On Fri, Aug 18, 2017 at 4:12 AM, Steve Loughran 
> wrote:

I can do a demo of this next week if people are interested

> On 17 Aug 2017, at 23:07, Aaron Fabbri 
> > wrote:
>
> Hello,
>
> I'd like to open a vote (7 days, ending August 24 at 3:10 PST) to merge the
> HADOOP-13345 feature branch into trunk.
>
> This branch contains the new S3Guard feature which adds metadata
> consistency features to the S3A client.  Formatted site documentation can
> be found here:
>
> https://github.com/apache/hadoop/blob/HADOOP-13345/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md
>
> The current patch against trunk is posted here:
>
> https://issues.apache.org/jira/browse/HADOOP-13998
>
> The branch modifies the s3a portion of the hadoop-tools/hadoop-aws module:
>
> - The feature is off by default, and care has been taken to insure it has
> no impact when disabled.
> - S3Guard can be enabled with the production database which is backed by
> DynamoDB, or with a local, in-memory implementation that facilitates
> integration testing without having to pay for a database.
> - getFileStatus() as well as directory listing consistency has been
> implemented and thoroughly tested, including delete tracking.
> - Convenient Maven profiles for testing with and without S3Guard.
> - New failure injection code and integration tests that exercise it.  We
> use timers and a wrapper around the Amazon SDK client object to force
> consistency delays to occur.  This allows us to assert that S3Guard works
> as advertised.  This will be extended with more types of failure injection
> to continue hardening the S3A client.
>
> Outside of hadoop-tools/hadoop-aws's s3a directory there are some minor
> changes:
>
> - core-default.xml defaults and documentation for s3guard parameters.
> - A couple additional FS contract test cases around rename.
> - More goodies in LambdaTestUtils
> - A new CLI tool for inspecting and manipulating S3Guard features,
> including the backing MetadataStore database.
>
> This branch has seen extensive testing as well as use in production.  This
> branch makes significant improvements to S3A's test toolkit as well.
>
> Performance is typically on par with, and in some cases better than, the
> existing S3A code without S3Guard enabled.
>
> This feature was developed with contributions and feedback from many
> people.  I'd like to thank everyone who worked on HADOOP-13345 as well as
> all of those who contributed feedback and work on the original design
> document.
>
> This is the first major Apache Hadoop project I've worked on from start to
> finish, and I've really enjoyed it.  Please shout if I've missed anything
> important here or in the VOTE process.
>
> Cheers,
> Aaron Fabbri


-
To unsubscribe, e-mail: 
common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: 
common-dev-h...@hadoop.apache.org




--
John

[jira] [Created] (HADOOP-14793) conftest command can't handle XInclude

[Steve Loughran (JIRA)]

Steve Loughran created HADOOP-14793:
---

 Summary: conftest command can't handle XInclude
 Key: HADOOP-14793
 URL: https://issues.apache.org/jira/browse/HADOOP-14793
 Project: Hadoop Common
  Issue Type: Bug
  Components: scripts, util
Affects Versions: 3.0.0-beta1
Reporter: Steve Loughran
Priority: Minor


hadoop conftest fails if there's an 

{code}
  http://www.w3.org/2001/XInclude;
href="something.xml">
  
{code}

expected: follows & validates the include. Actual: reports an error[.

{code}
 hadoop-dist/target/hadoop-3.0.0-beta1-SNAPSHOT/etc/hadoop/core-site.xml:
Line 23: element not 
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14719) Add StreamCapabilities support to WASB

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HADOOP-14719.
-
Resolution: Duplicate

> Add StreamCapabilities support to WASB
> --
>
> Key: HADOOP-14719
> URL: https://issues.apache.org/jira/browse/HADOOP-14719
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.0.0-alpha4
>Reporter: John Zhuge
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14790) PageBlobOutputStream to declare hflush/hsync StreamCapabilities for

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HADOOP-14790.
-
Resolution: Won't Fix

> PageBlobOutputStream to declare hflush/hsync StreamCapabilities for 
> 
>
> Key: HADOOP-14790
> URL: https://issues.apache.org/jira/browse/HADOOP-14790
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Priority: Minor
>
> HDFS-11644 added an interface for streams to export when dynamically 
> declaring support for features, as the static service APIs werent reliable. 
> As {{PageBlobOutputStream}} does support hsync/hflush, it should override 
> {{StreamCapabilities.hasCapability()}} and declare that it supports them.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14718) Add StreamCapabilities support to ADLS

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HADOOP-14718.
-
Resolution: Duplicate

> Add StreamCapabilities support to ADLS
> --
>
> Key: HADOOP-14718
> URL: https://issues.apache.org/jira/browse/HADOOP-14718
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha4
>Reporter: John Zhuge
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14792) Package on windows fail

[Ajith S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ajith S resolved HADOOP-14792.
--
Resolution: Not A Problem

as BUILDING.txt clearly states, cygwin is not supported, removing cygwin from 
PATH and using Git unix tools instead resolved this. marking as not a issue

> Package on windows fail
> ---
>
> Key: HADOOP-14792
> URL: https://issues.apache.org/jira/browse/HADOOP-14792
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Ajith S
>Assignee: Ajith S
> Attachments: packagefail.png
>
>
> {{mvn package -Pdist -Pnative-win -DskipTests -Dtar 
> -Dmaven.javadoc.skip=true}}
> command fails on windows 
> this is because 
> dev-support/bin/dist-copynativelibs need dos2unix conversion
> to avoid failure, we can add the conversion before bash execute



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-7793) mvn clean package -Dsrc does not work

[Andras Bokor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-7793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Bokor resolved HADOOP-7793.
--
Resolution: Invalid

> mvn clean package -Dsrc does not work
> -
>
> Key: HADOOP-7793
> URL: https://issues.apache.org/jira/browse/HADOOP-7793
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.23.0
>Reporter: Bruno Mahé
>  Labels: bigtop
>
> "mvn clean package -Dsrc" is supposed to create a source tarball in 
> hadoop-dist/target/ but some interactions with the clean target will prevent 
> this to happen.
> The following would happen:
> * -Dsrc makes maven create a source tarball in hadoop-dist/target/ for the 
> root module as well as each submodule
> 1. The root module would first create a source tarball correctly in 
> hadoop-dist/target/
> 2. Each submodule will also create a source tarball for their module in 
> /hadoop-dist/target/. But not before executing the clean target and 
> therefore deleting the main source tarball created at 1.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14792) Package on windows fail

[Ajith S (JIRA)]

Ajith S created HADOOP-14792:


 Summary: Package on windows fail
 Key: HADOOP-14792
 URL: https://issues.apache.org/jira/browse/HADOOP-14792
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Ajith S
Assignee: Ajith S


{{mvn package -Pdist -Pnative-win -DskipTests -Dtar -Dmaven.javadoc.skip=true}}
command fails on windows 
this is because 
dev-support/bin/dist-copynativelibs need dos2unix conversion

to avoid failure, we can add the conversion before bash execute



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org