Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/497/

[Aug 18, 2017 1:13:40 PM] (stevel) HADOOP-14769. WASB: delete recursive should 
not fail if a file is
[Aug 18, 2017 2:58:50 PM] (bibinchundatt) YARN-7007. NPE in RM while using 
YarnClient.getApplications().
[Aug 18, 2017 5:15:52 PM] (arp) Revert "HADOOP-14732. ProtobufRpcEngine should 
use Time.monotonicNow to
[Aug 18, 2017 9:38:44 PM] (yufei) YARN-6969. Clean up unused code in class 
FairSchedulerQueueInfo. (Larry
[Aug 19, 2017 1:26:36 AM] (wangda) YARN-6852. Native code changes to support 
isolate GPU devices by using




-1 overall


The following subsystems voted -1:
findbugs unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 
   Hard coded reference to an absolute pathname in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
 At DockerLinuxContainerRuntime.java:absolute pathname in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
 At DockerLinuxContainerRuntime.java:[line 490] 

Failed junit tests :

   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 
   hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 
   hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 
   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 
   
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation 
   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer 
   hadoop.yarn.sls.appmaster.TestAMSimulator 
   hadoop.yarn.sls.nodemanager.TestNMSimulator 

Timed out junit tests :

   
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA 
   org.apache.hadoop.yarn.client.api.impl.TestAMRMClient 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/497/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/497/artifact/out/diff-compile-javac-root.txt
  [296K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/497/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/497/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/497/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/497/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/497/artifact/out/whitespace-eol.txt
  [11M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/497/artifact/out/whitespace-tabs.txt
  [1.2M]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/497/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/497/artifact/out/diff-javadoc-javadoc-root.txt
  [1.9M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/497/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [320K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/497/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [64K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/497/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/497/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt
  [16K]

Powered by Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

[jira] [Created] (MAPREDUCE-6943) Mapreduce tasks for YARN Timeline Service v.2: beta 1

[Varun Saxena (JIRA)]

Varun Saxena created MAPREDUCE-6943:
---

 Summary: Mapreduce tasks for YARN Timeline Service v.2: beta 1
 Key: MAPREDUCE-6943
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6943
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Reporter: Varun Saxena
Assignee: Varun Saxena






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Branch merges and 3.0.0-beta1 scope

[Steve Loughran]

> On 19 Aug 2017, at 02:22, Andrew Wang  wrote:
> 
> * 3.0.0-beta1 includes S3Guard and YARN federation. Target date remains
> mid-Sept.
> * 3.0.0-GA includes TSv2 alpha2. Target date remains early November.
> * Everything else waits for 3.1, approximately March 2018.
> 


probably time to create those releases in JIRA, so people can start explicitly 
targeting them.


> On 19 Aug 2017, at 06:49, Vinod Kumar Vavilapalli  wrote:
> 
> Andrew,
> 
> Each of the branches below have been created more than a year ago (!) and 
> have been consistently worked upon and are now finally seeing the light of 
> the day. When they are "few weeks” away, pushing them out by 7 *more* months 
> just doesn’t make sense.
> 
> While I deeply appreciate the push for the dates, we shouldn’t be putting 
> moratorium on features like this till the proposed date is due. As a release 
> manager, it’s good to say that we will release a specific version as soon as 
> so-and-so features are in, but let’s not put exclusions like this.
> 
> I propose that, as we have done with every release so far, (and more 
> specifically the way we did 2.x alphas, betas, GA back in the day), we float 
> the date, let the individual branch contributors decide their fate. As long 
> as of course, they meet the timelines and the usual branch merge bar, testing 
> / compatibility / impact on rest of the code-base etc.
> 
> Anything short of that, we will just be incentivizing contributors away from 
> doing work in branches and towards putting stuff directly into trunk.
> 
> +Vinod


This is one of those curse-of-cadence things: The higher your release cadence, 
the less pressure to get "everything in". With a slower cadence, more pressure 
to get stuff in, more pressure to hold up the release, slows the cadence, gets 
even more stuff in, etc. etc.

- Andrew has been working on the release for months, we all need to appreciate 
how much hard work that is and has been, especially for what is going to be a 
major release.

- We know that things will be unstable in 3.0; Andrew's concern is about making 
sure that the newest, unstablest (?) features can at least be bypassed if there 
are problems. I we should also call out in the release notes what we think are 
the unstable bits where people need to use caution (example: S3Guard in 
"authoritative" mode)

- Anything related to wire compatibility has been problematic in the past; I 
think it's essential that whatever packets get sent around are going to be 
stable, so changes there need to be in, or at least the payloads set up ready 
for the features. Same for new public APIs.

- As fpr the rest, I don't know. I think being strict about it and ruthless in 
assessing the feature's stability & consequences of postponing the feature 
until a Hadoop 3.1 release in Jan/Feb, with a plan to ship then and follow up 
with a 3.2 in the summer.

Then: start planning that 3.1 release. Maybe I should put my hand up as release 
manager for that one. Then everyone would realise how amenable Andrew is being 
today.


One other thing: alongside the big branches, there's the eternal backlog of 
small patches. We should organise spending a few days updating, reviewing & 
merging them in

-Steve

Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk

[Rohith Sharma K S]

Hi Sangjin

Thanks for bringing this point.
We did similar exercise with current YARN-5355 branch today. Tests are
validated against default configuration and timeline server v.1.5 as well.

All YARN major features such as RM HA/Restart/work-preserving-restart, NM
restart, scheduling, sample MR/distributes shell jobs run are validated.
Everything looks fine.

This testing will also be done once YARN-5355 branch code freezed
completely as well in couple of days.

Thanks & Regards
Rohith Sharma K S

On 18 August 2017 at 22:56, Sangjin Lee  wrote:

> Kudos to Vrushali and the team for getting ready for this large and
> important feature! I know a huge team effort went into this. I look forward
> to seeing this merged.
>
> I'd like to ask one piece of due diligence. Could you please inspect
> rigorously to ensure that when disabled Timeline Service v.2 does not
> impact other features in any way? We did a similar exercise when we had the
> first drop, and it would be good to repeat that... Thanks!
>
> Sangjin
>
> On Wed, Aug 16, 2017 at 11:44 AM, Andrew Wang 
> wrote:
>
> > Great, thanks Vrushali! Sounds good to me.
> >
> > I have a few procedural release notes comments I'll put on YARN-5355, to
> > make sure we advertise this to our users appropriately.
> >
> > On Wed, Aug 16, 2017 at 11:32 AM, Vrushali Channapattan <
> > vrushal...@gmail.com> wrote:
> >
> > > Hi Andrew,
> > >
> > > Thanks for your response!
> > >
> > > There have been no changes to existing APIs since alpha1.
> > >
> > > We at Twitter have tested the feature to demonstrate it works at what
> we
> > > consider moderate scale but this did not include the security related
> > > testing. The security testing is in progress at present by Timeline
> > Service
> > > V2 team in the community and we think we will have more details on this
> > > very soon.
> > >
> > > About the jiras under YARN-5355: Only 3 of those sub-tasks are what we
> > > think of as "merge-blockers". The issues being targeted for merge are
> in
> > > [link1] below. There are about 59 jiras of which 56 are completed.
> > >
> > > We plan to make a new umbrella jira after the merge to trunk. We will
> > then
> > > create a new branch with the new jira name and move these open jiras
> > under
> > > YARN-5355 as subtasks of that new umbrella jira.
> > >
> > > thanks
> > > Vrushali
> > > [link1] https://issues.apache.org/jira/projects/YARN/versions/12337991
> > >
> > >
> > > On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang <
> andrew.w...@cloudera.com>
> > > wrote:
> > >
> > >> Hi Vrushali,
> > >>
> > >> Glad to hear this major dev milestone is nearing completion!
> > >>
> > >> Repeating my request on other merge [DISCUSS] threads, could you
> comment
> > >> on testing and API stability of this merge? Our timeline for beta1 is
> > about
> > >> a month out, so there's not much time to fix things beforehand.
> > >>
> > >> Looking at YARN-5355 there are also many unresolved subtasks. Should
> > most
> > >> of these be moved out to a new umbrella? I'm wondering what needs to
> be
> > >> completed before sending the merge vote.
> > >>
> > >> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex
> the
> > >> beta1 release date for this feature than others. Hopefully that won't
> be
> > >> necessary though :)
> > >>
> > >> Best,
> > >> Andrew
> > >>
> > >> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
> > >> vrushalic2...@gmail.com> wrote:
> > >>
> > >>> Looks like some of the hyperlinks appear messed up, my apologies,
> > >>> resending
> > >>> the same email with hopefully better looking content:
> > >>>
> > >>> Hi All,
> > >>>
> > >>> I'd like to open a discussion for merging Timeline Service v2
> > (YARN-5355)
> > >>> to trunk in a few weeks.
> > >>>
> > >>> We have previously completed one merge onto trunk [1] and Timeline
> > >>> Service
> > >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> > >>>
> > >>> Since then, we have been working on extending the capabilities of
> > >>> Timeline
> > >>> Service v2 in a feature branch [2].  There are a few related issues
> > >>> pending
> > >>> that are being actively worked upon and tested. As soon as they are
> > >>> resolved, we plan on starting a merge vote within the next two weeks.
> > The
> > >>> goal is to get this into hadoop3 beta.
> > >>>
> > >>> We have paid close attention to ensure that  once disabled Timeline
> > >>> Service
> > >>> v2 does not impact existing functionality when disabled (by default).
> > >>>
> > >>> At a high level, following are the key features that have been
> > >>> implemented
> > >>> since 3.0.0-alpha1:
> > >>> - Security (via Kerberos Authentication & delegation tokens)
> > [YARN-3053]
> > >>> - Timeline server usability improvements [timeline-server
> > >>>  > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> > >>> omponent%20%3D%20timelineserver%20AND%20labels%20!%3D%20atsv
> > >>> 2-hbase%20ORDER%20BY

Re: Map reduce sample program

[Remil Mohanan]

Hello Daniel,

I am trying to pass multiple non key values from mapper to reducer. I can pass 
key columns by concatenating the text key values. Can you please give a working 
sample code. Similarly for reading and writing a file inside the hdfs system 
other than normal read and write.

Thanks

Remil

Sent from my iPhone

> On 17 Aug 2017, at 12:34 AM, Daniel Templeton  wrote:
> 
> Can you clarify what you mean for #1?  For #2, try this:
> 
> https://tutorials.techmytalk.com/2014/08/16/hadoop-hdfs-java-api/
> 
> Daniel
> 
>> On 8/16/17 8:17 AM, Remil Mohanan wrote:
>> 
>> 
>>> 
>>> 
> Hi there,
> 
> Please help me to get a sample program for each scenario.
> 
> 1) need a Java map reducer sample program where multiple parameters
> are passed from mapper to reducer.
> 2) need a Java map reducer program where there is a write to a file inside
> hdfs filesystem as well as a read from a file inside hdfs other than
> the normal input file and output file mentioned in the mapper and reducer.
> 
> Have a nice day
> 
> Thanks
> 
> Remil
>> -
>> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>> 
> 

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org