date:20170922

[Update] Apache Hadoop 2.8.2 Release Status

2017-09-22 Thread Junping Du

Hi folks,
 I would like to give you a quick update on 2.8.2 release status:

- First release candidate (RC0) is published over the last weekend, but several 
docker container blockers (bugs, documents, etc.)
 are reported so we decided to cancel the RC0 for vote.

- New coming release blockers (for docker container support) are YARN-7034 
(just committed), YARN-6623, YARN-6930 and YARN-7230. 
Shane, Miklos and Varun are actively working on this. Appreciate the effort 
here!

- I will kick off new release candidate (RC1) once these blockers are resolved.

To all committers, branch-2.8.2 is still open for blocker/critical issues 
landing, but for major/minor/trivial issues, please commit to branch-2.8 and 
marked the fixed version as 2.8.3.

Thanks all for heads up. Have a good weekend!

Thanks,

Junping

From: Junping Du 
Sent: Tuesday, September 5, 2017 2:57 PM
To: larry mccay; Steve Loughran
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

I assume the quiet over the holiday means we agreed to move forward without 
taking HADOOP-14439 into 2.8.2.
There is a new release building (docker based) issue could be related to 
HADOOP-14474 where we removed oracle java 7 installer due to recent download 
address/contract change by Oracle. The build refuse to work - report as 
JAVA_HOME issue, but hard coded my local java home in create-release or 
Dockerfile doesn't help so we may need to add java 7 installation back (no 
matter Oracle JDK 7 or openJDK 7).
Filed HADOOP-14842 with more details to track as blocker for 2.8.2.

Thanks,

Junping

From: Junping Du 
Sent: Friday, September 1, 2017 12:37 PM
To: larry mccay; Steve Loughran
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

This issue (HADOOP-14439) is out of my radar given it is marked as Minor 
priority. If my understanding is correct, here is a trade-off between security 
and backward compatibility. IMO, priority of security is generally higher than 
backward compatibility especially 2.8.0 is still non-production release.
I think we should skip this for 2.8.2 in case it doesn't break compatibility 
from 2.7.x. Thoughts?

Thanks,

Junping

From: larry mccay 
Sent: Friday, September 1, 2017 10:55 AM
To: Steve Loughran
Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

If we do "fix" this in 2.8.2 we should seriously consider not doing so in
3.0.
This is a very poor practice.

I can see an argument for backward compatibility in 2.8.x line though.

On Fri, Sep 1, 2017 at 1:41 PM, Steve Loughran 
wrote:

> One thing we need to consider is
>
> HADOOP-14439: regression: secret stripping from S3x URIs breaks some
> downstream code
>
> Hadoop 2.8 has a best-effort attempt to strip out secrets from the
> toString() value of an s3a or s3n path where someone has embedded them in
> the URI; this has caused problems in some uses, specifically: when people
> use secrets this way (bad) and assume that you can round trip paths to
> string and back
>
> Should we fix this? If so, Hadoop 2.8.2 is the time to do it
>
>
> > On 1 Sep 2017, at 11:14, Junping Du  wrote:
> >
> > HADOOP-14814 get committed and HADOOP-9747 get push out to 2.8.3, so we
> are clean on blocker/critical issues now.
> > I finish practice of going through JACC report and no more incompatible
> public API changes get found between 2.8.2 and 2.7.4. Also I check commit
> history and fixed 10+ commits which are missing from branch-2.8.2 for some
> reason. So, the current branch-2.8.2 should be good to go for RC stage, and
> I will kick off our first RC tomorrow.
> > In the meanwhile, please don't land any commits to branch-2.8.2 since
> now. If some issues really belong to blocker, please ping me on the JIRA
> before doing any commits. branch-2.8 is still open for landing. Thanks for
> your cooperation!
> >
> >
> > Thanks,
> >
> > Junping
> >
> > 
> > From: Junping Du 
> > Sent: Wednesday, August 30, 2017 12:35 AM
> > To: Brahma Reddy Battula; common-...@hadoop.apache.org;
> hdfs-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org;
> yarn-dev@hadoop.apache.org
> > Subject: Re: Apache Hadoop 2.8.2 Release Plan
> >
> > Thanks Brahma for comment on this thread. To be clear, I always update
> branch version just before RC kicking off.
> >
> > For 2.8.2 release, I don't have plan to involve big top or other
> third-party test tools. As always, we will rely on

2017-09-22 Hadoop 3 release status update

2017-09-22 Thread Andrew Wang

https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3+release+status+updates

2017-09-22

We've had some late breaking blockers related to Docker support that are
delaying the release. We're on a day-by-day slip at this point.



Highlights:

   - I did a successful test create-release earlier this week.

Red flags:

   - Docker work resulted in some last minute blockers

Previously tracked beta1 blockers that have been resolved or dropped:

   - HADOOP-14771 
(hadoop-client
   does not include hadoop-yarn-client): Dropped this from the blocker list as
   it's mainly for documentation purposes
   - HDFS-12247 (Rename AddECPolicyResponse to
   AddErasureCodingPolicyResponse) was committed.

beta1 blockers:

   - YARN-6623  (Add
   support to turn off launching privileged containers in the
   container-executor): This is a newly escalated blocker related to the
   Docker work in YARN. Patch is up but we're still waiting on a commit.
   - HADOOP-14897  (Loosen
   compatibility guidelines for native dependencies): Raised by Chris Douglas,
   Daniel will post a patch soon.

beta1 features:

   - Erasure coding
  - Resolved last must-do for beta1!
  - People are looking more at the flaky tests and nice-to-haves
  - Eddy continues to make improvements to block reconstruction
  codepaths
   - Addressing incompatible changes (YARN-6142 and HDFS-11096)
   - Ray has gone through almost all the YARN protos and thinks we're okay
  to move forwards.
  - I think we'll move forward without this committed, given that Sean
  has run it successfully.
   - Classpath isolation (HADOOP-11656)
  - HADOOP-13917
 (Ensure
  nightly builds run the integration tests for the shaded client):
Sean wants
  to get this in before beta1 if there's time, it's already
catching issues.
  Relies on YETUS-543 which I reviewed, waiting on Allen.
  - HADOOP-14771 might be squeezed in if there's time.
   - Compat guide (HADOOP-13714
   )
  - HADOOP-14897 Above mentioned blocker filed by Chris Douglas.
   - TSv2 alpha 2
   - This was merged, no problems thus far [image: (smile)]

GA features:

   - Resource profiles (Wangda Tan)
  - Merge vote was sent out. Since branch-3.0 has been cut, this can be
  merged to trunk (3.1.0) and then backported once we've completed testing.
   - HDFS router-based federation (Chris Douglas)
   - This is like YARN federation, very separate and doesn't add new APIs,
  run in production at MSFT.
  - If it passes Cloudera internal integration testing, I'm fine
  putting this in for GA.
   - API-based scheduler configuration (Jonathan Hung)
  - Jonathan mentioned that his main goal is to get this in for 2.9.0,
  which seems likely to go out after 3.0.0 GA since there hasn't been any
  serious release planning yet. Jonathan said that delaying this
until 3.1.0
  is fine.
   - YARN native services
  - Still not 100% clear when this will land.

[jira] [Created] (YARN-7245) In Cap Sched UI, Max AM Resource column in Active Users Info section should be per-user

2017-09-22 Thread Eric Payne (JIRA)

Eric Payne created YARN-7245:


 Summary: In Cap Sched UI, Max AM Resource column in Active Users 
Info section should be per-user
 Key: YARN-7245
 URL: https://issues.apache.org/jira/browse/YARN-7245
 Project: Hadoop YARN
  Issue Type: Bug
  Components: capacity scheduler, yarn
Affects Versions: 3.0.0-alpha4, 2.8.1, 2.9.0
Reporter: Eric Payne






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] moving to Apache Yetus Audience Annotations

2017-09-22 Thread Sean Busbey

I'd refer to it as an incompatible change; we expressly label the
annotations as IA.Public.

If you think it's too late to get in for 3.0, I can make a jira and put it
on the back burner for when trunk goes to 4.0?

On Fri, Sep 22, 2017 at 12:49 PM, Andrew Wang 
wrote:

> Is this itself an incompatible change? I imagine the bytecode will be
> different.
>
> I think we're too late to do this for beta1 given that I want to cut an
> RC0 today.
>
> On Fri, Sep 22, 2017 at 7:03 AM, Sean Busbey  wrote:
>
>> When Apache Yetus formed, it started with several key pieces of Hadoop
>> that
>> looked reusable. In addition to our contribution testing infra, the
>> project
>> also stood up a version of our audience annotations for delineating the
>> public facing API[1].
>>
>> I recently got the Apache HBase community onto the Yetus version of those
>> annotations rather than their internal fork of the Hadoop ones[2]. It
>> wasn't pretty, mostly a lot of blind sed followed by spot checking and
>> reliance on automated tests.
>>
>> What do folks think about making the jump ourselves? I'd be happy to work
>> through things, either as one unreviewable monster or per-module
>> transitions (though a piece-meal approach might complicate our javadoc
>> situation).
>>
>>
>> [1]: http://yetus.apache.org/documentation/0.5.0/interface-classi
>> fication/
>> [2]: https://issues.apache.org/jira/browse/HBASE-17823
>>
>> --
>> busbey
>>
>
>


-- 
busbey

Re: [DISCUSS] moving to Apache Yetus Audience Annotations

2017-09-22 Thread Andrew Wang

Is this itself an incompatible change? I imagine the bytecode will be
different.

I think we're too late to do this for beta1 given that I want to cut an RC0
today.

On Fri, Sep 22, 2017 at 7:03 AM, Sean Busbey  wrote:

> When Apache Yetus formed, it started with several key pieces of Hadoop that
> looked reusable. In addition to our contribution testing infra, the project
> also stood up a version of our audience annotations for delineating the
> public facing API[1].
>
> I recently got the Apache HBase community onto the Yetus version of those
> annotations rather than their internal fork of the Hadoop ones[2]. It
> wasn't pretty, mostly a lot of blind sed followed by spot checking and
> reliance on automated tests.
>
> What do folks think about making the jump ourselves? I'd be happy to work
> through things, either as one unreviewable monster or per-module
> transitions (though a piece-meal approach might complicate our javadoc
> situation).
>
>
> [1]: http://yetus.apache.org/documentation/0.5.0/interface-classification/
> [2]: https://issues.apache.org/jira/browse/HBASE-17823
>
> --
> busbey
>

[jira] [Created] (YARN-7244) ShuffleHandler is not aware of disks that are added

2017-09-22 Thread Kuhu Shukla (JIRA)

Kuhu Shukla created YARN-7244:
-

 Summary: ShuffleHandler is not aware of disks that are added
 Key: YARN-7244
 URL: https://issues.apache.org/jira/browse/YARN-7244
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Kuhu Shukla
Assignee: Kuhu Shukla


The ShuffleHandler permanently remembers the list of "good" disks on NM 
startup. If disks later are added to the node then map tasks will start using 
them but the ShuffleHandler will not be aware of them. The end result is that 
the data cannot be shuffled from the node leading to fetch failures and re-runs 
of the map tasks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[DISCUSS] moving to Apache Yetus Audience Annotations

2017-09-22 Thread Sean Busbey

When Apache Yetus formed, it started with several key pieces of Hadoop that
looked reusable. In addition to our contribution testing infra, the project
also stood up a version of our audience annotations for delineating the
public facing API[1].

I recently got the Apache HBase community onto the Yetus version of those
annotations rather than their internal fork of the Hadoop ones[2]. It
wasn't pretty, mostly a lot of blind sed followed by spot checking and
reliance on automated tests.

What do folks think about making the jump ourselves? I'd be happy to work
through things, either as one unreviewable monster or per-module
transitions (though a piece-meal approach might complicate our javadoc
situation).


[1]: http://yetus.apache.org/documentation/0.5.0/interface-classification/
[2]: https://issues.apache.org/jira/browse/HBASE-17823

-- 
busbey

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2017-09-22 Thread Apache Jenkins Server

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/535/

[Sep 21, 2017 9:01:16 PM] (junping_du) YARN-7034. DefaultLinuxContainerRuntime 
and DockerLinuxContainerRuntime
[Sep 21, 2017 10:41:34 PM] (jlowe) YARN-4266. Allow users to enter containers 
as UID:GID pair instead of by
[Sep 22, 2017 2:37:04 AM] (szetszwo) HDFS-12507. StripedBlockUtil.java:694: 
warning - Tag @link: reference
[Sep 22, 2017 4:27:59 AM] (aajisaka) MAPREDUCE-6947. Moving logging APIs over 
to slf4j in
[Sep 22, 2017 6:07:59 AM] (aajisaka) MAPREDUCE-6966. DistSum should use 
Time.monotonicNow for measuring




-1 overall


The following subsystems voted -1:
docker


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
docker


Powered by Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] official docker image(s) for hadoop

2017-09-22 Thread Marton, Elek


Thanks all the feedbacks.

I created an issue:
https://issues.apache.org/jira/browse/HADOOP-14898

Let's continue the discussion there.

Thanks,
Marton

On 09/08/2017 02:45 PM, Marton, Elek wrote:


TL;DR: I propose to create official hadoop images and upload them to the 
dockerhub.


GOAL/SCOPE: I would like improve the existing documentation with 
easy-to-use docker based recipes to start hadoop clusters with various 
configuration.


The images also could be used to test experimental features. For example 
ozone could be tested easily with these compose file and configuration:


https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6

Or even the configuration could be included in the compose file:

https://github.com/elek/hadoop/blob/docker-2.8.0/example/docker-compose.yaml 



I would like to create separated example compose files for federation, 
ha, metrics usage, etc. to make it easier to try out and understand the 
features.


CONTEXT: There is an existing Jira 
https://issues.apache.org/jira/browse/HADOOP-13397
But it’s about a tool to generate production quality docker images 
(multiple types, in a flexible way). If no objections, I will create a 
separated issue to create simplified docker images for rapid prototyping 
and investigating new features. And register the branch to the dockerhub 
to create the images automatically.


MY BACKGROUND: I am working with docker based hadoop/spark clusters 
quite a while and run them succesfully in different environments 
(kubernetes, docker-swarm, nomad-based scheduling, etc.) My work is 
available from here: https://github.com/flokkr but they could handle 
more complex use cases (eg. instrumenting java processes with btrace, or 
read/reload configuration from consul).
  And IMHO in the official hadoop documentation it’s better to suggest 
to use official apache docker images and not external ones (which could 
be changed).


Please let me know if you have any comments.

Marton

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7243) Moving logging APIs over to slf4j in hadoop-yarn-server-resourcemanager

2017-09-22 Thread Yeliang Cang (JIRA)

Yeliang Cang created YARN-7243:
--

 Summary: Moving logging APIs over to slf4j in 
hadoop-yarn-server-resourcemanager
 Key: YARN-7243
 URL: https://issues.apache.org/jira/browse/YARN-7243
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Yeliang Cang
Assignee: Yeliang Cang






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2017-09-22 Thread Apache Jenkins Server

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/534/

No changes




-1 overall


The following subsystems voted -1:
unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting 
   hadoop.hdfs.server.namenode.TestReencryption 
   hadoop.yarn.server.nodemanager.containermanager.TestContainerManager 
   hadoop.yarn.server.resourcemanager.TestRMAdminService 
   
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation 
   hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 
   hadoop.yarn.server.router.webapp.TestRouterWebServiceUtil 
   hadoop.mapreduce.TestMapreduceConfigFields 
   hadoop.mapreduce.v2.hs.webapp.TestHSWebApp 
   hadoop.yarn.sls.TestSLSRunner 
   hadoop.yarn.sls.TestReservationSystemInvariants 

Timed out junit tests :

   org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands 
   
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA 
   
org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA 
   org.apache.hadoop.yarn.client.api.impl.TestAMRMClient 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/534/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/534/artifact/out/diff-compile-javac-root.txt
  [292K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/534/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/534/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/534/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/534/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/534/artifact/out/whitespace-eol.txt
  [11M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/534/artifact/out/whitespace-tabs.txt
  [1.2M]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/534/artifact/out/diff-javadoc-javadoc-root.txt
  [1.9M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/534/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [320K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/534/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [64K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/534/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [72K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/534/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [92K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/534/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
  [56K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/534/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt
  [472K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/534/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
  [24K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/534/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs.txt
  [28K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/534/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt
  [24K]

Powered by Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[Update] Apache Hadoop 2.8.2 Release Status

2017-09-22 Hadoop 3 release status update

[jira] [Created] (YARN-7245) In Cap Sched UI, Max AM Resource column in Active Users Info section should be per-user

Re: [DISCUSS] moving to Apache Yetus Audience Annotations

Re: [DISCUSS] moving to Apache Yetus Audience Annotations

[jira] [Created] (YARN-7244) ShuffleHandler is not aware of disks that are added

[DISCUSS] moving to Apache Yetus Audience Annotations

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

Re: [DISCUSS] official docker image(s) for hadoop

[jira] [Created] (YARN-7243) Moving logging APIs over to slf4j in hadoop-yarn-server-resourcemanager

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

11 matches

Site Navigation

Mail list logo

Footer information