Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/496/

[Aug 17, 2017 8:05:19 AM] (jzhuge) HADOOP-14560. Make HttpServer2 backlog size 
configurable. Contributed by
[Aug 17, 2017 9:37:15 AM] (sunilg) YARN-3254. HealthReport should include disk 
full information.
[Aug 17, 2017 4:35:36 PM] (wang) HDFS-12250. Reduce usage of 
FsPermissionExtension in unit tests.
[Aug 17, 2017 8:50:14 PM] (jlowe) YARN-6988. container-executor fails for 
docker when command length >
[Aug 17, 2017 10:26:11 PM] (wang) HDFS-12072. Provide fairness between EC and 
non-EC recovery tasks.
[Aug 17, 2017 11:23:48 PM] (manojpec) HDFS-12316. Verify HDFS snapshot deletion 
doesn't crash the ongoing file
[Aug 18, 2017 1:06:23 AM] (lei) HADOOP-14398. Modify documents for the 
FileSystem Builder API. (Lei




-1 overall


The following subsystems voted -1:
findbugs unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 
   Hard coded reference to an absolute pathname in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
 At DockerLinuxContainerRuntime.java:absolute pathname in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
 At DockerLinuxContainerRuntime.java:[line 490] 

Failed junit tests :

   hadoop.net.TestDNS 
   hadoop.hdfs.server.balancer.TestBalancer 
   
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation 
   hadoop.yarn.server.TestDiskFailures 
   hadoop.yarn.sls.appmaster.TestAMSimulator 
   hadoop.yarn.sls.nodemanager.TestNMSimulator 

Timed out junit tests :

   
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore 
   
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/496/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/496/artifact/out/diff-compile-javac-root.txt
  [296K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/496/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/496/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/496/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/496/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/496/artifact/out/whitespace-eol.txt
  [11M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/496/artifact/out/whitespace-tabs.txt
  [1.2M]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/496/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/496/artifact/out/diff-javadoc-javadoc-root.txt
  [1.9M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/496/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [148K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/496/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [232K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/496/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [64K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/496/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/496/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt
  [16K]

Powered by Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk

[Sangjin Lee]

Kudos to Vrushali and the team for getting ready for this large and
important feature! I know a huge team effort went into this. I look forward
to seeing this merged.

I'd like to ask one piece of due diligence. Could you please inspect
rigorously to ensure that when disabled Timeline Service v.2 does not
impact other features in any way? We did a similar exercise when we had the
first drop, and it would be good to repeat that... Thanks!

Sangjin

On Wed, Aug 16, 2017 at 11:44 AM, Andrew Wang 
wrote:

> Great, thanks Vrushali! Sounds good to me.
>
> I have a few procedural release notes comments I'll put on YARN-5355, to
> make sure we advertise this to our users appropriately.
>
> On Wed, Aug 16, 2017 at 11:32 AM, Vrushali Channapattan <
> vrushal...@gmail.com> wrote:
>
> > Hi Andrew,
> >
> > Thanks for your response!
> >
> > There have been no changes to existing APIs since alpha1.
> >
> > We at Twitter have tested the feature to demonstrate it works at what we
> > consider moderate scale but this did not include the security related
> > testing. The security testing is in progress at present by Timeline
> Service
> > V2 team in the community and we think we will have more details on this
> > very soon.
> >
> > About the jiras under YARN-5355: Only 3 of those sub-tasks are what we
> > think of as "merge-blockers". The issues being targeted for merge are in
> > [link1] below. There are about 59 jiras of which 56 are completed.
> >
> > We plan to make a new umbrella jira after the merge to trunk. We will
> then
> > create a new branch with the new jira name and move these open jiras
> under
> > YARN-5355 as subtasks of that new umbrella jira.
> >
> > thanks
> > Vrushali
> > [link1] https://issues.apache.org/jira/projects/YARN/versions/12337991
> >
> >
> > On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang 
> > wrote:
> >
> >> Hi Vrushali,
> >>
> >> Glad to hear this major dev milestone is nearing completion!
> >>
> >> Repeating my request on other merge [DISCUSS] threads, could you comment
> >> on testing and API stability of this merge? Our timeline for beta1 is
> about
> >> a month out, so there's not much time to fix things beforehand.
> >>
> >> Looking at YARN-5355 there are also many unresolved subtasks. Should
> most
> >> of these be moved out to a new umbrella? I'm wondering what needs to be
> >> completed before sending the merge vote.
> >>
> >> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex the
> >> beta1 release date for this feature than others. Hopefully that won't be
> >> necessary though :)
> >>
> >> Best,
> >> Andrew
> >>
> >> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
> >> vrushalic2...@gmail.com> wrote:
> >>
> >>> Looks like some of the hyperlinks appear messed up, my apologies,
> >>> resending
> >>> the same email with hopefully better looking content:
> >>>
> >>> Hi All,
> >>>
> >>> I'd like to open a discussion for merging Timeline Service v2
> (YARN-5355)
> >>> to trunk in a few weeks.
> >>>
> >>> We have previously completed one merge onto trunk [1] and Timeline
> >>> Service
> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> >>>
> >>> Since then, we have been working on extending the capabilities of
> >>> Timeline
> >>> Service v2 in a feature branch [2].  There are a few related issues
> >>> pending
> >>> that are being actively worked upon and tested. As soon as they are
> >>> resolved, we plan on starting a merge vote within the next two weeks.
> The
> >>> goal is to get this into hadoop3 beta.
> >>>
> >>> We have paid close attention to ensure that  once disabled Timeline
> >>> Service
> >>> v2 does not impact existing functionality when disabled (by default).
> >>>
> >>> At a high level, following are the key features that have been
> >>> implemented
> >>> since 3.0.0-alpha1:
> >>> - Security (via Kerberos Authentication & delegation tokens)
> [YARN-3053]
> >>> - Timeline server usability improvements [timeline-server
> >>>  >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> >>> omponent%20%3D%20timelineserver%20AND%20labels%20!%3D%20atsv
> >>> 2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%
> >>> 2C%20created%20ASC>
> >>> ]
> >>> - HBase specific improvements [atsv2-hbase
> >>>  >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20l
> >>> abels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
> >>> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> >>> ]
> >>> - REST API additions and improvements [timeline-reader
> >>>  >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> >>> omponent%20in%20(timelineclient%2C%20timelinereader)%
> >>> 20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> >>> ]
> >>> - Reader side simple authorization via whitelist [YARN-6820]
> >>>
> >

Re: [DISCUSS] Merge YARN resource profile (YARN-3926) branch into trunk

[Andrew Wang]

Hi Wangda,

Can this feature be disabled? Is it on or off by default? We're 1 month
from the target release for beta1, so I don't want to introduce risk to
existing code paths. TSv2 and S3Guard and YARN Federation are all okay in
that regard.

I'm also not clear on what work is remaining, there are a lot of unresolved
subtasks still under YARN-3926.

In terms of compatibility, the design doc talks about some PB changes. Are
these stable? Is there documentation somewhere that explains what APIs are
stable or unstable for users?

I'm going to start a separate discussion about beta1 scope. I think this is
the fourth merge proposal this week, and this amount of code movement makes
me very nervous considering that beta1 is only a month out.

Best,
Andrew

On Thu, Aug 17, 2017 at 8:27 PM, Wangda Tan  wrote:

> +hdfs/common/mr
>
> On Thu, Aug 17, 2017 at 1:28 PM, Wangda Tan  wrote:
>
> > Hi all,
> >
> > I want to hear your thoughts of merging YARN resource profile branch into
> > trunk in the next few weeks. The goal is to get it in for Hadoop 3.0
> beta1.
> >
> > *Regarding to testing:*
> > We did extensive tests for the feature in the last several months.
> > Comparing to latest trunk.
> > - For SLS benchmark: We didn't see observable performance gap from
> > simulated test based on 8K nodes SLS traces (1 PB memory). We got 3k+
> > containers allocated per second.
> > - For microbenchmark: We use performance test cases added by YARN-6775,
> it
> > shows around 5% performance regression comparing to trunk.
> >
> > *Regarding to API stability: *
> > Most new added @Public APIs are @Unstable (We're going to convert some
> new
> > added @Public/@Evolving to @Unstable in the cleanup JIRA as well), we
> want
> > to get this included by beta1 so we get some feedbacks before declaring
> > stable API.
> >
> > There're few pending cleanups under YARN-3926 umbrella JIRA. Besides
> these
> > cleanups, this feature works from end-to-end, we will do another
> iteration
> > of end-to-end tests after cleanup patches got committed.
> >
> > We would love to get your thoughts before opening a voting thread.
> >
> > Special thanks to a team of folks who worked hard and contributed towards
> > this efforts including design discussion / patch / reviews, etc.: Varun
> > Vasudev, Sunil Govind, Daniel Templeton, Vinod Vavilapalli, Yufei Gu,
> > Karthik Kambatla, Jason Lowe, Arun Suresh.
> >
> > Thanks,
> > Wangda Tan
> >
>

Re: [DISCUSS] Merge yarn-native-services branch into trunk

[Andrew Wang]

Hi Jian, thanks for the reply,

On Thu, Aug 17, 2017 at 1:03 PM, Jian He  wrote:

> Thanks Andrew for the comments. Answers below:
>
> - There are no new APIs added in YARN/Hadoop core. In fact, all the new
> code are running outside of existing system and they are optional and
> require users to explicitly opt in. The new system’s own rest API is not
> stable and will be evolving.
>

Great! That adds a lot more confidence that this is safe to merge.

Are these new APIs listed in user documentation, and described as unstable?


> - We have been running/testing a version of the entire system internally
> for quite a while.
>

Do you mind elaborating on the level of testing? Number of nodes, types of
applications, production or test workload, etc. It'd help us build
confidence.


> - I’d like to see this in hadoop3-beta1. Of course, we’ll take
> responsibility of moving fast and not block the potential timeline.
>

Few more questions:

How should we advertise this feature in the release? Since the APIs are
unstable, I'd propose calling it "alpha" in the release notes, like we do
the TSv2.

Could you move out subtasks from YARN-5079 that are not blocking the merge?
This would make it easier to understand what's remaining.

Thanks,
Andrew

Re: [DISCUSS] Merge YARN resource profile (YARN-3926) branch into trunk

[Wangda Tan]

Hi Andrew,

This feature can be disabled, It is off by default. The major change of
existing code path it Resource YARN PB record related implementations, we
did lots of tests around this regarding to performances and safety of these
changes, so far so good (please refer to my email regarding to performance
and compatibility). Beyond resource PB implementation changes, it mostly
add new code path instead of modifying existing logics. We will continue to
do more verifications next week to minimize risks.

As I mentioned, new added fields are marked as Unstable right now (and we
will convert some @evolving to  @unstable). They're all stated in javadocs.

I completely understand the recent emerging merge request makes everybody
nervous, we will try to do more tests and move fast to meet the time line.
:)

Please let me know if you have any other concerns.

Thanks,
Wangda


On Fri, Aug 18, 2017 at 2:34 PM, Andrew Wang 
wrote:

> Hi Wangda,
>
> Can this feature be disabled? Is it on or off by default? We're 1 month
> from the target release for beta1, so I don't want to introduce risk to
> existing code paths. TSv2 and S3Guard and YARN Federation are all okay in
> that regard.
>
> I'm also not clear on what work is remaining, there are a lot of
> unresolved subtasks still under YARN-3926.
>
> In terms of compatibility, the design doc talks about some PB changes. Are
> these stable? Is there documentation somewhere that explains what APIs are
> stable or unstable for users?
>
> I'm going to start a separate discussion about beta1 scope. I think this
> is the fourth merge proposal this week, and this amount of code movement
> makes me very nervous considering that beta1 is only a month out.
>
> Best,
> Andrew
>
> On Thu, Aug 17, 2017 at 8:27 PM, Wangda Tan  wrote:
>
>> +hdfs/common/mr
>>
>> On Thu, Aug 17, 2017 at 1:28 PM, Wangda Tan  wrote:
>>
>> > Hi all,
>> >
>> > I want to hear your thoughts of merging YARN resource profile branch
>> into
>> > trunk in the next few weeks. The goal is to get it in for Hadoop 3.0
>> beta1.
>> >
>> > *Regarding to testing:*
>> > We did extensive tests for the feature in the last several months.
>> > Comparing to latest trunk.
>> > - For SLS benchmark: We didn't see observable performance gap from
>> > simulated test based on 8K nodes SLS traces (1 PB memory). We got 3k+
>> > containers allocated per second.
>> > - For microbenchmark: We use performance test cases added by YARN-6775,
>> it
>> > shows around 5% performance regression comparing to trunk.
>> >
>> > *Regarding to API stability: *
>> > Most new added @Public APIs are @Unstable (We're going to convert some
>> new
>> > added @Public/@Evolving to @Unstable in the cleanup JIRA as well), we
>> want
>> > to get this included by beta1 so we get some feedbacks before declaring
>> > stable API.
>> >
>> > There're few pending cleanups under YARN-3926 umbrella JIRA. Besides
>> these
>> > cleanups, this feature works from end-to-end, we will do another
>> iteration
>> > of end-to-end tests after cleanup patches got committed.
>> >
>> > We would love to get your thoughts before opening a voting thread.
>> >
>> > Special thanks to a team of folks who worked hard and contributed
>> towards
>> > this efforts including design discussion / patch / reviews, etc.: Varun
>> > Vasudev, Sunil Govind, Daniel Templeton, Vinod Vavilapalli, Yufei Gu,
>> > Karthik Kambatla, Jason Lowe, Arun Suresh.
>> >
>> > Thanks,
>> > Wangda Tan
>> >
>>
>
>

Branch merges and 3.0.0-beta1 scope

[Andrew Wang]

Hi folks,

As you might have seen, we've had a number of branch merges floated this
past week targeted for 3.0.0-beta1, which is planned for about a month from
now.

In total, I'm currently tracking these branches:

YARN-2915: YARN federation (recently merged)
HADOOP-13345: S3Guard (currently being voted on)
YARN-5355: TSv2 alpha2 ("few weeks")
YARN-5079: Native services ("few weeks")
YARN-3926: Resource profiles ("few weeks")

We should effectively be in code freeze (only blockers/criticals), so the
volume of merge proposals at this point came as a surprise. Despite our
best efforts as software engineers, big code movement always comes with
risk.

Since we started the 3.0 release series close to a year ago, I'm also loath
to increase the scope. The main motivation for 3.0 was to deliver JDK8 and
HDFS EC, and our users deserve a production-quality release with these
features. We've also been good about the release cadence thus far in 3.x,
so a 3.1 isn't that far out.

Here's my proposal:

* 3.0.0-beta1 includes S3Guard and YARN federation. Target date remains
mid-Sept.
* 3.0.0-GA includes TSv2 alpha2. Target date remains early November.
* Everything else waits for 3.1, approximately March 2018.

My rationale for inclusion and exclusion is as follows:

Inclusion:

* YARN federation has been run in production, does not touch existing code,
adds no new APIs, and is off by default.
* S3Guard has been run in production and is off by default.
* The first iteration of TSv2 was shipped in 3.0.0-alpha1, so we're
committed to this for 3.0.0 GA. It's off by default and adds no impact.

Exclusion:

* The primary reason for exclusion is to maintain the planned release
schedule. Native services and resource profiles are still a few weeks from
being ready for merge.
* A reminder that 3.1 is only another 3 months after 3.0 given our release
cadence thus far. If there's demand, we could even do a 3.1 immediately
following 3.0.

I'm happy to talk with the contributors of each of these features to
understand their timelines and requirements, with the caveat that I'll be
out through Wednesday next week. Please reach out.

Best,
Andrew

Re: Branch merges and 3.0.0-beta1 scope

[Vinod Kumar Vavilapalli]

Andrew,

Each of the branches below have been created more than a year ago (!) and have 
been consistently worked upon and are now finally seeing the light of the day. 
When they are "few weeks” away, pushing them out by 7 *more* months just 
doesn’t make sense.

While I deeply appreciate the push for the dates, we shouldn’t be putting 
moratorium on features like this till the proposed date is due. As a release 
manager, it’s good to say that we will release a specific version as soon as 
so-and-so features are in, but let’s not put exclusions like this.

I propose that, as we have done with every release so far, (and more 
specifically the way we did 2.x alphas, betas, GA back in the day), we float 
the date, let the individual branch contributors decide their fate. As long as 
of course, they meet the timelines and the usual branch merge bar, testing / 
compatibility / impact on rest of the code-base etc.

Anything short of that, we will just be incentivizing contributors away from 
doing work in branches and towards putting stuff directly into trunk.

+Vinod

> On Aug 18, 2017, at 6:22 PM, Andrew Wang  wrote:
> 
> Hi folks,
> 
> As you might have seen, we've had a number of branch merges floated this
> past week targeted for 3.0.0-beta1, which is planned for about a month from
> now.
> 
> In total, I'm currently tracking these branches:
> 
> YARN-2915: YARN federation (recently merged)
> HADOOP-13345: S3Guard (currently being voted on)
> YARN-5355: TSv2 alpha2 ("few weeks")
> YARN-5079: Native services ("few weeks")
> YARN-3926: Resource profiles ("few weeks")
> 
> We should effectively be in code freeze (only blockers/criticals), so the
> volume of merge proposals at this point came as a surprise. Despite our
> best efforts as software engineers, big code movement always comes with
> risk.
> 
> Since we started the 3.0 release series close to a year ago, I'm also loath
> to increase the scope. The main motivation for 3.0 was to deliver JDK8 and
> HDFS EC, and our users deserve a production-quality release with these
> features. We've also been good about the release cadence thus far in 3.x,
> so a 3.1 isn't that far out.
> 
> Here's my proposal:
> 
> * 3.0.0-beta1 includes S3Guard and YARN federation. Target date remains
> mid-Sept.
> * 3.0.0-GA includes TSv2 alpha2. Target date remains early November.
> * Everything else waits for 3.1, approximately March 2018.
> 
> My rationale for inclusion and exclusion is as follows:
> 
> Inclusion:
> 
> * YARN federation has been run in production, does not touch existing code,
> adds no new APIs, and is off by default.
> * S3Guard has been run in production and is off by default.
> * The first iteration of TSv2 was shipped in 3.0.0-alpha1, so we're
> committed to this for 3.0.0 GA. It's off by default and adds no impact.
> 
> Exclusion:
> 
> * The primary reason for exclusion is to maintain the planned release
> schedule. Native services and resource profiles are still a few weeks from
> being ready for merge.
> * A reminder that 3.1 is only another 3 months after 3.0 given our release
> cadence thus far. If there's demand, we could even do a 3.1 immediately
> following 3.0.
> 
> I'm happy to talk with the contributors of each of these features to
> understand their timelines and requirements, with the caveat that I'll be
> out through Wednesday next week. Please reach out.
> 
> Best,
> Andrew


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org