[jira] [Created] (HDFS-12446) FSNamesystem#internalReleaseLease throw IllegalStateException

2017-09-13 Thread Jiandan Yang (JIRA) 
Jiandan Yang  created HDFS-12446:


 Summary: FSNamesystem#internalReleaseLease throw 
IllegalStateException
 Key: HDFS-12446
 URL: https://issues.apache.org/jira/browse/HDFS-12446
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 2.8.1
Reporter: Jiandan Yang 



2017-09-14 10:21:32,042 INFO 
org.apache.hadoop.hdfs.server.namenode.LeaseManager: [Lease.  Holder: 
DFSClient_NONMAPREDUCE_-275421369_84, pending creates: 7] has expired hard limit
2017-09-14 10:21:32,042 INFO 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Recovering [Lease.  
Holder: DFSClient_NONMAPREDUCE_-275421369_84, pending creates: 7], 
src=/user/ads/af_base_n_adf_p4p_pv/data/55f57d72-1542-4acf-b2d4-08af65b0e859
2017-09-14 10:21:32,042 WARN 
org.apache.hadoop.hdfs.server.namenode.LeaseManager: Unexpected throwable:
java.lang.IllegalStateException: Unexpected block state: 
blk_1265519060_203004758 is COMMITTED but not COMPLETE, 
file=55f57d72-1542-4acf-b2d4-08af65b0e859 (INodeFile), 
blocks=[blk_1265519060_203004758] (i=0)
at 
com.google.common.base.Preconditions.checkState(Preconditions.java:172)
at 
org.apache.hadoop.hdfs.server.namenode.INodeFile.assertAllBlocksComplete(INodeFile.java:218)
at 
org.apache.hadoop.hdfs.server.namenode.INodeFile.toCompleteFile(INodeFile.java:207)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.finalizeINodeFileUnderConstruction(FSNamesystem.java:3312)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.internalReleaseLease(FSNamesystem.java:3184)
at 
org.apache.hadoop.hdfs.server.namenode.LeaseManager.checkLeases(LeaseManager.java:383)
at 
org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:329)
at java.lang.Thread.run(Thread.java:834)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12445) Spelling mistakes in the Hadoop java source files viz. choosen instead of chosen.

2017-09-13 Thread hu xiaodong (JIRA) 
hu xiaodong created HDFS-12445:
--

 Summary: Spelling mistakes in the Hadoop java source files viz. 
choosen instead of chosen.
 Key: HDFS-12445
 URL: https://issues.apache.org/jira/browse/HDFS-12445
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: hu xiaodong
Assignee: hu xiaodong
Priority: Trivial


I found spelling mistakes in the Hadoop java source files viz. choosen instead 
of chosen.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12444) Reduce runtime of TestWriteReadStripedFile

2017-09-13 Thread Andrew Wang (JIRA) 
Andrew Wang created HDFS-12444:
--

 Summary: Reduce runtime of TestWriteReadStripedFile
 Key: HDFS-12444
 URL: https://issues.apache.org/jira/browse/HDFS-12444
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: erasure-coding, test
Affects Versions: 3.0.0-alpha4
Reporter: Andrew Wang
Assignee: Andrew Wang


This test takes a long time to run since it writes a lot of data, and 
frequently times out during precommit testing. If we change the EC policy from 
RS(6,3) to RS(3,2) then it will run a lot faster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSS] official docker image(s) for hadoop

2017-09-13 Thread Mingliang Liu 
> It would be very helpful for testing the RC.
For testing and voting, I have been using docker containers for a while, see 
code at: https://github.com/weiqingy/caochong 


> TL;DR: I propose to create official hadoop images and upload them to the 
> dockerhub
I’m +1 on this idea. The “official” docker image basically means a commitment 
to maintain well documented and broadly tested images, which seems not a burden 
to us.

Ceph has a community docker project https://github.com/ceph/ceph-docker 
 and I think our scope here is similar to 
it.

Mingliang

> On Sep 13, 2017, at 11:39 AM, Yufei Gu  wrote:
> 
> It would be very helpful for testing the RC. To vote a RC, committers and
> PMCs usually spend lots of time to compile, deploy the RC, do several
> sanity tests, then +1 for the RC. The docker image potentially saves the
> compilation and deployment time, and people can do more tests.
> 
> Best,
> 
> Yufei
> 
> On Wed, Sep 13, 2017 at 11:19 AM, Wangda Tan  wrote:
> 
>> +1 to add Hadoop docker image for easier testing / prototyping, it gonna be
>> super helpful!
>> 
>> Thanks,
>> Wangda
>> 
>> On Wed, Sep 13, 2017 at 10:48 AM, Miklos Szegedi <
>> miklos.szeg...@cloudera.com> wrote:
>> 
>>> Marton, thank you for working on this. I think Official Docker images for
>>> Hadoop would be very useful for a lot of reasons. I think that it is
>> better
>>> to have a coordinated effort with production ready base images with
>>> dependent images for prototyping. Does anyone else have an opinion about
>>> this?
>>> 
>>> Thank you,
>>> Miklos
>>> 
>>> On Fri, Sep 8, 2017 at 5:45 AM, Marton, Elek  wrote:
>>> 
 
 TL;DR: I propose to create official hadoop images and upload them to
>> the
 dockerhub.
 
 GOAL/SCOPE: I would like improve the existing documentation with
 easy-to-use docker based recipes to start hadoop clusters with various
 configuration.
 
 The images also could be used to test experimental features. For
>> example
 ozone could be tested easily with these compose file and configuration:
 
 https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6
 
 Or even the configuration could be included in the compose file:
 
 https://github.com/elek/hadoop/blob/docker-2.8.0/example/doc
 ker-compose.yaml
 
 I would like to create separated example compose files for federation,
>>> ha,
 metrics usage, etc. to make it easier to try out and understand the
 features.
 
 CONTEXT: There is an existing Jira https://issues.apache.org/jira
 /browse/HADOOP-13397
 But it’s about a tool to generate production quality docker images
 (multiple types, in a flexible way). If no objections, I will create a
 separated issue to create simplified docker images for rapid
>> prototyping
 and investigating new features. And register the branch to the
>> dockerhub
>>> to
 create the images automatically.
 
 MY BACKGROUND: I am working with docker based hadoop/spark clusters
>> quite
 a while and run them succesfully in different environments (kubernetes,
 docker-swarm, nomad-based scheduling, etc.) My work is available from
>>> here:
 https://github.com/flokkr but they could handle more complex use cases
 (eg. instrumenting java processes with btrace, or read/reload
>>> configuration
 from consul).
 And IMHO in the official hadoop documentation it’s better to suggest
>> to
 use official apache docker images and not external ones (which could be
 changed).
 
 Please let me know if you have any comments.
 
 Marton
 
 -
 To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
 For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
 
 
>>> 
>>

[jira] [Created] (HDFS-12443) Ozone: Improve SCM block deletion throttling algorithm

2017-09-13 Thread Weiwei Yang (JIRA) 
Weiwei Yang created HDFS-12443:
--

 Summary: Ozone: Improve SCM block deletion throttling algorithm 
 Key: HDFS-12443
 URL: https://issues.apache.org/jira/browse/HDFS-12443
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone, scm
Reporter: Weiwei Yang
Assignee: Weiwei Yang


Currently SCM scans delLog to send deletion transactions to datanode 
periodically, the throttling algorithm is simple, it scans at most 
{{BLOCK_DELETE_TX_PER_REQUEST_LIMIT}} (by default 50) at a time. This is 
non-optimal, worst case it might cache 50 TXs for 50 different DNs so each DN 
will only get 1 TX to proceed in an interval, this will make the deletion slow. 
An improvement to this is to make this throttling by datanode, e.g 50 TXs per 
datanode per interval.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12442) WebHdfsFileSystem#getFileBlockLocations will always return BlockLocation#corrupt as false

2017-09-13 Thread Rushabh S Shah (JIRA) 
Rushabh S Shah created HDFS-12442:
-

 Summary: WebHdfsFileSystem#getFileBlockLocations will always 
return BlockLocation#corrupt as false
 Key: HDFS-12442
 URL: https://issues.apache.org/jira/browse/HDFS-12442
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 3.0.0-alpha2, 2.9.0
Reporter: Rushabh S Shah
Assignee: Rushabh S Shah
Priority: Critical


Was going through {{JsonUtilClient#toBlockLocation}}
Below is the relevant code snippet.
{code:title=JsonUtilClient.java|borderStyle=solid}
 /** Convert a Json map to BlockLocation. **/
  static BlockLocation toBlockLocation(Map m)
  throws IOException{
...
...  
boolean corrupt = Boolean.
getBoolean(m.get("corrupt").toString());
...
...
  }
{code}
According to java docs for {{Boolean#getBoolean}}
{noformat}
Returns true if and only if the system property named by the argument exists 
and is equal to the string "true". 
{noformat}
I assume, the map value for key {{corrupt}} will be populated with either 
{{true}} or {{false}}.
On the client side, {{Boolean#getBoolean}} will look for system property for 
true or false.
So it will always return false unless the system property is set for true or 
false.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSS] official docker image(s) for hadoop

2017-09-13 Thread Yufei Gu 
It would be very helpful for testing the RC. To vote a RC, committers and
PMCs usually spend lots of time to compile, deploy the RC, do several
sanity tests, then +1 for the RC. The docker image potentially saves the
compilation and deployment time, and people can do more tests.

Best,

Yufei

On Wed, Sep 13, 2017 at 11:19 AM, Wangda Tan  wrote:

> +1 to add Hadoop docker image for easier testing / prototyping, it gonna be
> super helpful!
>
> Thanks,
> Wangda
>
> On Wed, Sep 13, 2017 at 10:48 AM, Miklos Szegedi <
> miklos.szeg...@cloudera.com> wrote:
>
> > Marton, thank you for working on this. I think Official Docker images for
> > Hadoop would be very useful for a lot of reasons. I think that it is
> better
> > to have a coordinated effort with production ready base images with
> > dependent images for prototyping. Does anyone else have an opinion about
> > this?
> >
> > Thank you,
> > Miklos
> >
> > On Fri, Sep 8, 2017 at 5:45 AM, Marton, Elek  wrote:
> >
> > >
> > > TL;DR: I propose to create official hadoop images and upload them to
> the
> > > dockerhub.
> > >
> > > GOAL/SCOPE: I would like improve the existing documentation with
> > > easy-to-use docker based recipes to start hadoop clusters with various
> > > configuration.
> > >
> > > The images also could be used to test experimental features. For
> example
> > > ozone could be tested easily with these compose file and configuration:
> > >
> > > https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6
> > >
> > > Or even the configuration could be included in the compose file:
> > >
> > > https://github.com/elek/hadoop/blob/docker-2.8.0/example/doc
> > > ker-compose.yaml
> > >
> > > I would like to create separated example compose files for federation,
> > ha,
> > > metrics usage, etc. to make it easier to try out and understand the
> > > features.
> > >
> > > CONTEXT: There is an existing Jira https://issues.apache.org/jira
> > > /browse/HADOOP-13397
> > > But it’s about a tool to generate production quality docker images
> > > (multiple types, in a flexible way). If no objections, I will create a
> > > separated issue to create simplified docker images for rapid
> prototyping
> > > and investigating new features. And register the branch to the
> dockerhub
> > to
> > > create the images automatically.
> > >
> > > MY BACKGROUND: I am working with docker based hadoop/spark clusters
> quite
> > > a while and run them succesfully in different environments (kubernetes,
> > > docker-swarm, nomad-based scheduling, etc.) My work is available from
> > here:
> > > https://github.com/flokkr but they could handle more complex use cases
> > > (eg. instrumenting java processes with btrace, or read/reload
> > configuration
> > > from consul).
> > >  And IMHO in the official hadoop documentation it’s better to suggest
> to
> > > use official apache docker images and not external ones (which could be
> > > changed).
> > >
> > > Please let me know if you have any comments.
> > >
> > > Marton
> > >
> > > -
> > > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> > >
> > >
> >
>

Re: [DISCUSS] official docker image(s) for hadoop

2017-09-13 Thread Bharat Viswanadham 
+1 (non-binding)
It would be really nice to have Docker to try different features of Hadoop 
(like HA, Federation Enabled, Erasure coding…), which will helpful for both 
developers and users.


Thanks,
Bharat


On 9/13/17, 11:31 AM, "Eric Badger"  wrote:

+1 definitely think an official Hadoop docker image (possibly 1 per major
or minor release) would be a positive both for contributors and for users
of Hadoop.

Eric

On Wed, Sep 13, 2017 at 1:19 PM, Wangda Tan  wrote:

> +1 to add Hadoop docker image for easier testing / prototyping, it gonna 
be
> super helpful!
>
> Thanks,
> Wangda
>
> On Wed, Sep 13, 2017 at 10:48 AM, Miklos Szegedi <
> miklos.szeg...@cloudera.com> wrote:
>
> > Marton, thank you for working on this. I think Official Docker images 
for
> > Hadoop would be very useful for a lot of reasons. I think that it is
> better
> > to have a coordinated effort with production ready base images with
> > dependent images for prototyping. Does anyone else have an opinion about
> > this?
> >
> > Thank you,
> > Miklos
> >
> > On Fri, Sep 8, 2017 at 5:45 AM, Marton, Elek  wrote:
> >
> > >
> > > TL;DR: I propose to create official hadoop images and upload them to
> the
> > > dockerhub.
> > >
> > > GOAL/SCOPE: I would like improve the existing documentation with
> > > easy-to-use docker based recipes to start hadoop clusters with various
> > > configuration.
> > >
> > > The images also could be used to test experimental features. For
> example
> > > ozone could be tested easily with these compose file and 
configuration:
> > >
> > > https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6
> > >
> > > Or even the configuration could be included in the compose file:
> > >
> > > https://github.com/elek/hadoop/blob/docker-2.8.0/example/doc
> > > ker-compose.yaml
> > >
> > > I would like to create separated example compose files for federation,
> > ha,
> > > metrics usage, etc. to make it easier to try out and understand the
> > > features.
> > >
> > > CONTEXT: There is an existing Jira https://issues.apache.org/jira
> > > /browse/HADOOP-13397
> > > But it’s about a tool to generate production quality docker images
> > > (multiple types, in a flexible way). If no objections, I will create a
> > > separated issue to create simplified docker images for rapid
> prototyping
> > > and investigating new features. And register the branch to the
> dockerhub
> > to
> > > create the images automatically.
> > >
> > > MY BACKGROUND: I am working with docker based hadoop/spark clusters
> quite
> > > a while and run them succesfully in different environments 
(kubernetes,
> > > docker-swarm, nomad-based scheduling, etc.) My work is available from
> > here:
> > > https://github.com/flokkr but they could handle more complex use cases
> > > (eg. instrumenting java processes with btrace, or read/reload
> > configuration
> > > from consul).
> > >  And IMHO in the official hadoop documentation it’s better to suggest
> to
> > > use official apache docker images and not external ones (which could 
be
> > > changed).
> > >
> > > Please let me know if you have any comments.
> > >
> > > Marton
> > >
> > > -
> > > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> > >
> > >
> >
>

Re: [DISCUSS] official docker image(s) for hadoop

2017-09-13 Thread Eric Badger 
+1 definitely think an official Hadoop docker image (possibly 1 per major
or minor release) would be a positive both for contributors and for users
of Hadoop.

Eric

On Wed, Sep 13, 2017 at 1:19 PM, Wangda Tan  wrote:

> +1 to add Hadoop docker image for easier testing / prototyping, it gonna be
> super helpful!
>
> Thanks,
> Wangda
>
> On Wed, Sep 13, 2017 at 10:48 AM, Miklos Szegedi <
> miklos.szeg...@cloudera.com> wrote:
>
> > Marton, thank you for working on this. I think Official Docker images for
> > Hadoop would be very useful for a lot of reasons. I think that it is
> better
> > to have a coordinated effort with production ready base images with
> > dependent images for prototyping. Does anyone else have an opinion about
> > this?
> >
> > Thank you,
> > Miklos
> >
> > On Fri, Sep 8, 2017 at 5:45 AM, Marton, Elek  wrote:
> >
> > >
> > > TL;DR: I propose to create official hadoop images and upload them to
> the
> > > dockerhub.
> > >
> > > GOAL/SCOPE: I would like improve the existing documentation with
> > > easy-to-use docker based recipes to start hadoop clusters with various
> > > configuration.
> > >
> > > The images also could be used to test experimental features. For
> example
> > > ozone could be tested easily with these compose file and configuration:
> > >
> > > https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6
> > >
> > > Or even the configuration could be included in the compose file:
> > >
> > > https://github.com/elek/hadoop/blob/docker-2.8.0/example/doc
> > > ker-compose.yaml
> > >
> > > I would like to create separated example compose files for federation,
> > ha,
> > > metrics usage, etc. to make it easier to try out and understand the
> > > features.
> > >
> > > CONTEXT: There is an existing Jira https://issues.apache.org/jira
> > > /browse/HADOOP-13397
> > > But it’s about a tool to generate production quality docker images
> > > (multiple types, in a flexible way). If no objections, I will create a
> > > separated issue to create simplified docker images for rapid
> prototyping
> > > and investigating new features. And register the branch to the
> dockerhub
> > to
> > > create the images automatically.
> > >
> > > MY BACKGROUND: I am working with docker based hadoop/spark clusters
> quite
> > > a while and run them succesfully in different environments (kubernetes,
> > > docker-swarm, nomad-based scheduling, etc.) My work is available from
> > here:
> > > https://github.com/flokkr but they could handle more complex use cases
> > > (eg. instrumenting java processes with btrace, or read/reload
> > configuration
> > > from consul).
> > >  And IMHO in the official hadoop documentation it’s better to suggest
> to
> > > use official apache docker images and not external ones (which could be
> > > changed).
> > >
> > > Please let me know if you have any comments.
> > >
> > > Marton
> > >
> > > -
> > > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> > >
> > >
> >
>

Re: [DISCUSS] official docker image(s) for hadoop

2017-09-13 Thread Wangda Tan 
+1 to add Hadoop docker image for easier testing / prototyping, it gonna be
super helpful!

Thanks,
Wangda

On Wed, Sep 13, 2017 at 10:48 AM, Miklos Szegedi <
miklos.szeg...@cloudera.com> wrote:

> Marton, thank you for working on this. I think Official Docker images for
> Hadoop would be very useful for a lot of reasons. I think that it is better
> to have a coordinated effort with production ready base images with
> dependent images for prototyping. Does anyone else have an opinion about
> this?
>
> Thank you,
> Miklos
>
> On Fri, Sep 8, 2017 at 5:45 AM, Marton, Elek  wrote:
>
> >
> > TL;DR: I propose to create official hadoop images and upload them to the
> > dockerhub.
> >
> > GOAL/SCOPE: I would like improve the existing documentation with
> > easy-to-use docker based recipes to start hadoop clusters with various
> > configuration.
> >
> > The images also could be used to test experimental features. For example
> > ozone could be tested easily with these compose file and configuration:
> >
> > https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6
> >
> > Or even the configuration could be included in the compose file:
> >
> > https://github.com/elek/hadoop/blob/docker-2.8.0/example/doc
> > ker-compose.yaml
> >
> > I would like to create separated example compose files for federation,
> ha,
> > metrics usage, etc. to make it easier to try out and understand the
> > features.
> >
> > CONTEXT: There is an existing Jira https://issues.apache.org/jira
> > /browse/HADOOP-13397
> > But it’s about a tool to generate production quality docker images
> > (multiple types, in a flexible way). If no objections, I will create a
> > separated issue to create simplified docker images for rapid prototyping
> > and investigating new features. And register the branch to the dockerhub
> to
> > create the images automatically.
> >
> > MY BACKGROUND: I am working with docker based hadoop/spark clusters quite
> > a while and run them succesfully in different environments (kubernetes,
> > docker-swarm, nomad-based scheduling, etc.) My work is available from
> here:
> > https://github.com/flokkr but they could handle more complex use cases
> > (eg. instrumenting java processes with btrace, or read/reload
> configuration
> > from consul).
> >  And IMHO in the official hadoop documentation it’s better to suggest to
> > use official apache docker images and not external ones (which could be
> > changed).
> >
> > Please let me know if you have any comments.
> >
> > Marton
> >
> > -
> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >
> >
>

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2017-09-13 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/522/

[Sep 12, 2017 4:19:09 PM] (wangda) YARN-4081. Add support for multiple resource 
types in the Resource
[Sep 12, 2017 4:19:09 PM] (wangda) YARN-4172. Extend DominantResourceCalculator 
to account for all
[Sep 12, 2017 4:19:10 PM] (wangda) YARN-4715. Add support to read resource 
types from a config file.
[Sep 12, 2017 4:19:10 PM] (wangda) YARN-4829. Add support for binary units in 
Resource class.(vvasudev via
[Sep 12, 2017 4:19:10 PM] (wangda) YARN-4830. Add support for resource types in 
the nodemanager.
[Sep 12, 2017 4:19:10 PM] (wangda) YARN-5242. Update DominantResourceCalculator 
to consider all resource
[Sep 12, 2017 4:19:10 PM] (wangda) YARN-5586. Update the Resources class to 
consider all resource types.
[Sep 12, 2017 4:19:10 PM] (wangda) YARN-5707. Add manager class for resource 
profiles. Contributed by Varun
[Sep 12, 2017 4:19:10 PM] (wangda) YARN-5708. Implement APIs to get resource 
profiles from the RM.
[Sep 12, 2017 4:19:10 PM] (wangda) YARN-5587. Add support for resource 
profiles. (vvasudev via asuresh)
[Sep 12, 2017 4:19:11 PM] (wangda) YARN-5588. [YARN-3926] Add support for 
resource profiles in distributed
[Sep 12, 2017 4:19:11 PM] (wangda) YARN-6232. Update resource usage and 
preempted resource calculations to
[Sep 12, 2017 4:19:11 PM] (wangda) YARN-6445. [YARN-3926] Performance 
improvements in resource profile
[Sep 12, 2017 4:19:11 PM] (wangda) YARN-6761. Fix build for YARN-3926 branch. 
Contributed by Varun Vasudev.
[Sep 12, 2017 4:19:11 PM] (wangda) YARN-6786. [YARN-3926] ResourcePBImpl 
imports cleanup. Contributed by
[Sep 12, 2017 4:19:11 PM] (wangda) YARN-6788. [YARN-3926] Improve performance 
of resource profile branch
[Sep 12, 2017 4:19:11 PM] (wangda) YARN-6935. [YARN-3926] 
ResourceProfilesManagerImpl.parseResource() has
[Sep 12, 2017 4:19:11 PM] (wangda) YARN-6994. [YARN-3926] Remove last uses of 
Long from resource types
[Sep 12, 2017 4:19:12 PM] (wangda) YARN-6892. [YARN-3926] Improve API 
implementation in Resources and
[Sep 12, 2017 4:19:12 PM] (wangda) YARN-6908. ResourceProfilesManagerImpl is 
missing @Overrides on methods
[Sep 12, 2017 4:19:12 PM] (wangda) YARN-6610. [YARN-3926] 
DominantResourceCalculator#getResourceAsValue
[Sep 12, 2017 4:19:12 PM] (wangda) YARN-7030. [YARN-3926] Performance 
optimizations in Resource and
[Sep 12, 2017 4:19:12 PM] (wangda) YARN-7042. Clean up unit tests after 
YARN-6610. (Daniel Templeton via
[Sep 12, 2017 4:19:12 PM] (wangda) YARN-6789. Add Client API to get all 
supported resource types from RM.
[Sep 12, 2017 4:19:12 PM] (wangda) YARN-6781. [YARN-3926] 
ResourceUtils#initializeResourcesMap takes an
[Sep 12, 2017 4:19:12 PM] (wangda) YARN-7043. Cleanup ResourceProfileManager. 
(wangda)
[Sep 12, 2017 4:19:12 PM] (wangda) YARN-7067. [YARN-3926] Optimize ResourceType 
information display in UI.
[Sep 12, 2017 4:19:12 PM] (wangda) YARN-7039. Fix javac and javadoc errors in 
YARN-3926 branch. (Sunil G
[Sep 12, 2017 4:19:12 PM] (wangda) YARN-7093. Improve log message in 
ResourceUtils. (Sunil G via wangda)
[Sep 12, 2017 4:19:12 PM] (wangda) YARN-6933. [YARN-3926] 
ResourceUtils.DISALLOWED_NAMES check is
[Sep 12, 2017 4:19:12 PM] (wangda) YARN-7056. Document Resource Profiles 
feature. (Sunil G via wangda)
[Sep 12, 2017 4:19:12 PM] (wangda) YARN-7136. Additional Performance 
Improvement for Resource Profile
[Sep 12, 2017 4:19:12 PM] (wangda) YARN-7137. [YARN-3926] Move newly added APIs 
to unstable in YARN-3926
[Sep 12, 2017 5:04:22 PM] (rchiang) HADOOP-14798. Update sshd-core and related 
mina-core library versions.
[Sep 12, 2017 5:19:34 PM] (rchiang) HADOOP-14799. Update nimbus-jose-jwt to 
4.41.1. (rchiang)
[Sep 12, 2017 5:36:04 PM] (rchiang) HADOOP-14796. Update json-simple version to 
1.1.1. (rchiang)
[Sep 12, 2017 5:53:48 PM] (rchiang) HADOOP-14648. Bump commons-configuration2 
to 2.1.1. (rchiang)
[Sep 12, 2017 6:12:44 PM] (rchiang) HADOOP-14653. Update joda-time version to 
2.9.9. (rchiang)
[Sep 12, 2017 6:20:56 PM] (wang) HDFS-12417. Disable flaky 
TestDFSStripedOutputStreamWithFailure.
[Sep 12, 2017 6:35:21 PM] (rchiang) HADOOP-14797. Update re2j version to 1.1. 
(rchiang)
[Sep 12, 2017 8:37:38 PM] (rchiang) HADOOP-14856. Fix AWS, Jetty, HBase, 
Ehcache entries for NOTICE.txt.
[Sep 12, 2017 9:51:08 PM] (jlowe) HADOOP-14843. Improve FsPermission symbolic 
parsing unit test coverage.
[Sep 12, 2017 11:10:08 PM] (Arun Suresh) YARN-7185. ContainerScheduler should 
only look at availableResource for
[Sep 12, 2017 11:13:39 PM] (yufei) YARN-7057. FSAppAttempt#getResourceUsage 
doesn't need to consider
[Sep 12, 2017 11:18:41 PM] (arp) HDFS-12407. Journal node fails to shutdown 
cleanly if
[Sep 13, 2017 12:03:32 AM] (Arun Suresh) YARN-7185. [Addendum patch] Minor 
javadoc and checkstyle fix.
[Sep 13, 2017 12:35:30 AM] (wang) HDFS-1. Document and test BlockLocation 
for erasure-coded files.
[Sep 13, 2017 1:12:07 AM] (lei) HDFS-12412. Change

Re: [VOTE] Merge yarn-native-services branch into trunk

2017-09-13 Thread Allen Wittenauer 

> On Sep 8, 2017, at 9:25 AM, Jian He  wrote:
> 
> Hi Allen,
> The documentations are committed. Please check QuickStart.md and others in 
> the same folder.
> YarnCommands.md doc is updated to include new commands.
> DNS default port is also documented. 
> Would you like to give a look and see if it address your concerns ?

Somewhat. Greatly improved, but there’s still way too much “we’re 
working on this” and “here’s a link to a JIRA” and just general brokenness 
going on.

Here’s some examples from concepts.  Concepts!  The document I’d expect 
to give me very basic “when we talk about X, we mean Y” definitions:

"A host of scheduling features are being developed to support long running 
services.”

Yeah, ok?  How is this a concept?

  or

"[YARN-3998](https://issues.apache.org/jira/browse/YARN-3998) 
implements a retry-policy to let NM re-launch a service container when it 
fails.”


The patch itself went through nine revisions and a long discussion. 
Would an end user care about the details in that JIRA?  

If the answer to the last question is YES, then the documentation has 
failed.  The whole point of documentation is so they don’t have to go digging 
into the details of the implementation, the decision process that got us there, 
etc.  If they care enough about the details, they’ll run through the changelog 
and click on the JIRA link there.  If the summary line of the changelog isn’t 
obvious, well… then we need better summaries.

etc, etc.

...

The sleep example is nice.  Now, let’s see a non-toy example:  multiple 
instances of Apache httpd or MariaDB or something real and not from the Hadoop 
echo chamber (e.g., non-JVM-based).  If this is for “native” services, this 
shouldn’t be a problem, right?  Give a real example and users will buy what 
you’re selling.  I also think writing the docs and providing an example of 
doing something big and outside the team’s comfort zone will clarify where end 
users are going to need more help than what’s being provided.  Getting a 
MariaDB instance or three up will help tremendously here.

Which reminds me: something the documentation doesn’t cover is storage. 
What happens to it, where does it come from, etc, etc.  That’s an important 
detail that I didn’t see covered.  (I may have missed it.)  

…

Why are there directions to enable other, partially unrelated services 
in here?  Shouldn’t there be pointers to their specific documentation?  Is the 
expectation that if the requirements for those other services change that 
contributors will need to update multiple documents?

"Start the DNS server”

Just… yikes.

a) yarn classname … This is not how we do user-facing things. 
The fact it’s not really possible for a *daemon* to be put in the 
YarnCommands.md doc should be a giant red flag that something isn’t going 
correctly here.
b) no jsvc support for something that it’s strongly hinted at 
wanting to run privileged = an instant -1 for failing basic security practices. 
 There’s zero reason for it to be running continually as root.
c) If this would have been hooked into the shell scripts 
appropriately, logs, user switching, etc would have been had for free.
d) Where’s stop?  Right. Since it’s outside the scripts, there 
is no pid support so one has to do all of that manually….


Given:

 "3. Supports reverse lookups (name based on IP). Note, this works only 
for Docker containers.”

then:

"It should not be used as a fully-functional corporate DNS.”

Scratch corporate.  It’s not a fully functional DNS server if it can’t do 
reverse lookups.  (Which, ironically, means it’s not suitable for use with 
Apache Hadoop, given it requires both fwd and rev DNS ...)



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12441) Suppress UnresolvedPathException in namenode log

2017-09-13 Thread Kihwal Lee (JIRA) 
Kihwal Lee created HDFS-12441:
-

 Summary: Suppress UnresolvedPathException in namenode log
 Key: HDFS-12441
 URL: https://issues.apache.org/jira/browse/HDFS-12441
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Kihwal Lee
Priority: Minor


{{UnresolvedPathException}} as a normal process of resolving symlinks. This 
doesn't need to be logged at all.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-12440) Ozone: TestAllocateContainer fails on jenkins

2017-09-13 Thread Weiwei Yang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved HDFS-12440.

   Resolution: Duplicate
Fix Version/s: HDFS-7240

> Ozone: TestAllocateContainer fails on jenkins
> -
>
> Key: HDFS-12440
> URL: https://issues.apache.org/jira/browse/HDFS-12440
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Minor
> Fix For: HDFS-7240
>
>
> I am seeing this failure in [this jenkins 
> report|https://builds.apache.org/job/PreCommit-HDFS-Build/21089/testReport/org.apache.hadoop.ozone.scm/TestAllocateContainer/testAllocate/],
>  with following error
> {noformat}
> Stacktrace
> java.lang.NullPointerException
>  at 
> org.apache.hadoop.ozone.scm.node.SCMNodeManager.getNodeStat(SCMNodeManager.java:828)
>  at 
> org.apache.hadoop.ozone.scm.container.placement.algorithms.SCMCommonPolicy.hasEnoughSpace(SCMCommonPolicy.java:147)
>  at 
> org.apache.hadoop.ozone.scm.container.placement.algorithms.SCMCommonPolicy.lambda$chooseDatanodes$0(SCMCommonPolicy.java:125)
>  at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
>  at 
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
>  at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
>  at 
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
>  at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
>  at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
>  at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
>  at 
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12440) Ozone: TestAllocateContainer fails on jenkins

2017-09-13 Thread Weiwei Yang (JIRA) 
Weiwei Yang created HDFS-12440:
--

 Summary: Ozone: TestAllocateContainer fails on jenkins
 Key: HDFS-12440
 URL: https://issues.apache.org/jira/browse/HDFS-12440
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone
Affects Versions: HDFS-7240
Reporter: Weiwei Yang
Assignee: Weiwei Yang
Priority: Minor


I am seeing this failure in [this jenkins 
report|https://builds.apache.org/job/PreCommit-HDFS-Build/21067/testReport/org.apache.hadoop.ozone.scm/TestAllocateContainer/org_apache_hadoop_ozone_scm_TestAllocateContainer/],
 with following error

{noformat}
Stacktrace

java.lang.NullPointerException: null
at 
org.apache.hadoop.hdfs.MiniDFSCluster.setDataNodeStorageCapacities(MiniDFSCluster.java:1715)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.setDataNodeStorageCapacities(MiniDFSCluster.java:1694)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1674)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:882)
at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:494)
at 
org.apache.hadoop.ozone.MiniOzoneCluster.(MiniOzoneCluster.java:98)
at 
org.apache.hadoop.ozone.MiniOzoneCluster.(MiniOzoneCluster.java:77)
at 
org.apache.hadoop.ozone.MiniOzoneCluster$Builder.build(MiniOzoneCluster.java:441)
at 
org.apache.hadoop.ozone.scm.TestAllocateContainer.init(TestAllocateContainer.java:56)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org