date:20180912

[jira] [Created] (HADOOP-15754) s3guard: testDynamoTableTagging should clear existing config

2018-09-12 Thread Aaron Fabbri (JIRA)

Aaron Fabbri created HADOOP-15754:
-

 Summary: s3guard: testDynamoTableTagging should clear existing 
config
 Key: HADOOP-15754
 URL: https://issues.apache.org/jira/browse/HADOOP-15754
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/s3
Affects Versions: 3.2.0
Reporter: Aaron Fabbri


I recently committed HADOOP-14734 which adds support for tagging Dynamo DB 
tables for S3Guard when they are created.

 

Later, when testing another patch, I hit a test failure because I still had a 
tag option set in my test configuration (auth-keys.xml) that was adding my own 
table tag.
{noformat}
[ERROR] 
testDynamoTableTagging(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB)
  Time elapsed: 13.384 s  <<< FAILURE!

java.lang.AssertionError: expected:<2> but was:<3>

        at org.junit.Assert.fail(Assert.java:88)

        at org.junit.Assert.failNotEquals(Assert.java:743)

        at org.junit.Assert.assertEquals(Assert.java:118)

        at org.junit.Assert.assertEquals(Assert.java:555)

        at org.junit.Assert.assertEquals(Assert.java:542)

        at 
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB.testDynamoTableTagging(ITestS3GuardToolDynamoDB.java:129)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)

        at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)

        at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)

        at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)

        at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)

        at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)

        at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)

        at 
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74){noformat}
I think the solution is just to clear any tag.* options set in the 
configuration at the beginning of the test.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15753) ABFS: support path "abfs://mycluster/file/path"

2018-09-12 Thread Da Zhou (JIRA)

Da Zhou created HADOOP-15753:


 Summary: ABFS: support path "abfs://mycluster/file/path"
 Key: HADOOP-15753
 URL: https://issues.apache.org/jira/browse/HADOOP-15753
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Da Zhou
Assignee: Da Zhou


WASB support path format: "wasb://mycluster/file/path", but ABFS doesn't, which 
caused some issues for customer. I will add support for this path format.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.5 (RC0)

2018-09-12 Thread Kitti Nánási

Thanks for working on this Junping!

+1 (non-binding)

- checked out git tag release-2.8.5-RC0
- built from source on Mac OS X 10.13.4, java version 8.0.172-zulu
- deployed on a 3 node cluster
- ran terasort, teragen, teravalidate with success
- executed basic hdfs and dfsadmin commands

Thanks,
Kitti


On Wed, Sep 12, 2018 at 2:37 PM, Yesha Vora  wrote:

> +1 (binding)
>
> Tested the following:
> - launch Pseudo-Distributed 1 node cluster
> - Tested with word count / teragen / terasort / nnbench/ mrbench on yarn
> - Validate yarn UI
> - Test basic yarn cli operations such as -kill, -status, -appStates,
> -appTypes
>
> --
> Yesha Vora
>
>
> 
> From: 俊平堵 
> Sent: Monday, September 10, 2018 5:00 AM
> To: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org;
> yarn-...@hadoop.apache.org
> Cc: junpingdu(堵俊平)
> Subject: [VOTE] Release Apache Hadoop 2.8.5 (RC0)
>
> Hi all,
>
>  I've created the first release candidate (RC0) for Apache
> Hadoop 2.8.5. This is our next point release to follow up 2.8.4. It
> includes 33 important fixes and improvements.
>
>
> The RC artifacts are available at:
> http://home.apache.org/~junping_du/hadoop-2.8.5-RC0
>
>
> The RC tag in git is: release-2.8.5-RC0
>
>
>
> The maven artifacts are available via repository.apache.org<
> http://repository.apache.org> at:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1140
>
>
> Please try the release and vote; the vote will run for the usual 5
> working
> days, ending on 9/15/2018 PST time.
>
>
> Thanks,
>
>
> Junping
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 2.8.5 (RC0)

2018-09-12 Thread Yesha Vora

+1 (binding)

Tested the following:
- launch Pseudo-Distributed 1 node cluster
- Tested with word count / teragen / terasort / nnbench/ mrbench on yarn 
- Validate yarn UI
- Test basic yarn cli operations such as -kill, -status, -appStates, -appTypes

--
Yesha Vora



From: 俊平堵 
Sent: Monday, September 10, 2018 5:00 AM
To: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org
Cc: junpingdu(堵俊平)
Subject: [VOTE] Release Apache Hadoop 2.8.5 (RC0)

Hi all,

 I've created the first release candidate (RC0) for Apache
Hadoop 2.8.5. This is our next point release to follow up 2.8.4. It
includes 33 important fixes and improvements.


The RC artifacts are available at:
http://home.apache.org/~junping_du/hadoop-2.8.5-RC0


The RC tag in git is: release-2.8.5-RC0



The maven artifacts are available via repository.apache.org<
http://repository.apache.org> at:

https://repository.apache.org/content/repositories/orgapachehadoop-1140


Please try the release and vote; the vote will run for the usual 5 working
days, ending on 9/15/2018 PST time.


Thanks,


Junping


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [Vote] Merge discussion for Node attribute support feature YARN-3409

2018-09-12 Thread Shane Kumpf

I just committed YARN-8768 to trunk. Thanks for the report and quick fix.

Thanks,
-Shane

On Wed, Sep 12, 2018 at 2:28 PM Naganarasimha Garla <
naganarasimha...@apache.org> wrote:

> Thanks for reporting Sean and Sunil for acting on it. My bad i had used
> alias to compile which was skipping the javadoc !
> Will try to commit as soon as the jenkins results are out.
>
> On Thu, Sep 13, 2018 at 1:18 AM Sunil G  wrote:
>
> > Thanks Sean
> >
> > YARN-8768 is raised and we will get this in very soon.
> >
> > - Sunil
> >
> > On Wed, Sep 12, 2018 at 9:16 PM Sean Mackrory 
> > wrote:
> >
> >> `mvn install` fails on trunk right now due to javadoc error in yarn-api.
> >> Is
> >> that related? If so, can we revert or fix ASAP please?
> >>
> >> On Wed, Sep 12, 2018 at 6:05 AM Naganarasimha Garla <
> >> naganarasimha...@apache.org> wrote:
> >>
> >> > Hi All,
> >> >  Me and Sunil have successfully merged YARN-3409 to the trunk
> >> and
> >> > we have created new umbrella jira YARN-8766 to track the pending jiras
> >> and
> >> > the remaining planned improvement features.
> >> > Thanks all for the support !
> >> >
> >> > Regards,
> >> > + Naga
> >> >
> >> >
> >> > On Wed, Sep 12, 2018 at 7:46 AM Naganarasimha Garla <
> >> > naganarasimha...@apache.org> wrote:
> >> >
> >> > > Hi All,
> >> > >  Voting has been running since 6 days and adding my vote we
> >> have
> >> > 4
> >> > > binding and 2 non binding +1's with no -1's this voting passes and
> we
> >> > will
> >> > > be merging the branch shortly. Thanks for all who participated in
> the
> >> > > discussion and voting thread !
> >> > >
> >> > > Thanks and Regards,
> >> > > + Naga
> >> > >
> >> > > On Mon, Sep 10, 2018 at 2:50 PM Zian Chen 
> >> wrote:
> >> > >
> >> > >> +1 for merge.
> >> > >>
> >> > >> > On Sep 9, 2018, at 10:47 PM, Weiwei Yang 
> >> wrote:
> >> > >> >
> >> > >> > +1 for the merge
> >> > >> >
> >> > >> > On Mon, Sep 10, 2018 at 12:06 PM Rohith Sharma K S <
> >> > >> > rohithsharm...@apache.org> wrote:
> >> > >> >
> >> > >> >> +1 for merge
> >> > >> >>
> >> > >> >> -Rohith Sharma K S
> >> > >> >>
> >> > >> >> On Wed, 5 Sep 2018 at 18:01, Naganarasimha Garla <
> >> > >> >> naganarasimha...@apache.org> wrote:
> >> > >> >>
> >> > >> >>> Hi All,
> >> > >> >>>Thanks for feedback folks, based on the positive
> response
> >> > >> >> starting
> >> > >> >>> a Vote thread for merging YARN-3409 to master.
> >> > >> >>>
> >> > >> >>> Regards,
> >> > >> >>> + Naga & Sunil
> >> > >> >>>
> >> > >> >>> On Wed, 5 Sep 2018 2:51 am Wangda Tan, 
> >> wrote:
> >> > >> >>>
> >> > >>  +1 for the merge, it gonna be a great addition to 3.2.0
> release.
> >> > >> Thanks
> >> > >> >>> to
> >> > >>  everybody for pushing this feature to complete.
> >> > >> 
> >> > >>  Best,
> >> > >>  Wangda
> >> > >> 
> >> > >>  On Tue, Sep 4, 2018 at 8:25 AM Bibinchundatt <
> >> > >> >> bibin.chund...@huawei.com>
> >> > >>  wrote:
> >> > >> 
> >> > >> > +1 for merge. Fetaure would be a good addition to 3.2
> release.
> >> > >> >
> >> > >> > --
> >> > >> > Bibin A Chundatt
> >> > >> > M: +91-9742095715 <+91%2097420%2095715>
> >> > >> > E: bibin.chund...@huawei.com bibin.chund...@huawei.com>
> >> > >> > 2012实验室-印研IT&Cloud BU分部
> >> > >> > 2012 Laboratories-IT&Cloud BU Branch Dept.
> >> > >> > From:Naganarasimha Garla
> >> > >> > To:common-dev@hadoop.apache.org,Hdfs-dev,
> >> > yarn-...@hadoop.apache.org
> >> > >> ,
> >> > >> > mapreduce-...@hadoop.apache.org,
> >> > >> > Date:2018-08-29 20:00:44
> >> > >> > Subject:[Discuss] Merge discussion for Node attribute support
> >> > >> feature
> >> > >> > YARN-3409
> >> > >> >
> >> > >> > Hi All,
> >> > >> >
> >> > >> > We would like to hear your thoughts on merging “Node
> Attributes
> >> > >> >> Support
> >> > >> >>> in
> >> > >> > YARN” branch (YARN-3409) [2] into trunk in a few weeks. The
> >> goal
> >> > is
> >> > >> to
> >> > >> >>> get
> >> > >> > it in for HADOOP 3.2.
> >> > >> >
> >> > >> > *Major work happened in this branch*
> >> > >> >
> >> > >> > YARN-6858. Attribute Manager to store and provide node
> >> attributes
> >> > in
> >> > >> >> RM
> >> > >> > YARN-7871. Support Node attributes reporting from NM to RM(
> >> > >> >> distributed
> >> > >> > node attributes)
> >> > >> > YARN-7863. Modify placement constraints to support node
> >> attributes
> >> > >> > YARN-7875. Node Attribute store for storing and recovering
> >> > >> attributes
> >> > >> >
> >> > >> > *Detailed Design:*
> >> > >> >
> >> > >> > Please refer [1] for detailed design document.
> >> > >> >
> >> > >> > *Testing Efforts:*
> >> > >> >
> >> > >> > We did detailed tests for the feature in the last few weeks.
> >> > >> > This feature will be enabled only when Node Attributes
> >> constraints
> >> > >> are
> >> > >> > sp

Re: [Vote] Merge discussion for Node attribute support feature YARN-3409

2018-09-12 Thread Naganarasimha Garla

Thanks for reporting Sean and Sunil for acting on it. My bad i had used
alias to compile which was skipping the javadoc !
Will try to commit as soon as the jenkins results are out.

On Thu, Sep 13, 2018 at 1:18 AM Sunil G  wrote:

> Thanks Sean
>
> YARN-8768 is raised and we will get this in very soon.
>
> - Sunil
>
> On Wed, Sep 12, 2018 at 9:16 PM Sean Mackrory 
> wrote:
>
>> `mvn install` fails on trunk right now due to javadoc error in yarn-api.
>> Is
>> that related? If so, can we revert or fix ASAP please?
>>
>> On Wed, Sep 12, 2018 at 6:05 AM Naganarasimha Garla <
>> naganarasimha...@apache.org> wrote:
>>
>> > Hi All,
>> >  Me and Sunil have successfully merged YARN-3409 to the trunk
>> and
>> > we have created new umbrella jira YARN-8766 to track the pending jiras
>> and
>> > the remaining planned improvement features.
>> > Thanks all for the support !
>> >
>> > Regards,
>> > + Naga
>> >
>> >
>> > On Wed, Sep 12, 2018 at 7:46 AM Naganarasimha Garla <
>> > naganarasimha...@apache.org> wrote:
>> >
>> > > Hi All,
>> > >  Voting has been running since 6 days and adding my vote we
>> have
>> > 4
>> > > binding and 2 non binding +1's with no -1's this voting passes and we
>> > will
>> > > be merging the branch shortly. Thanks for all who participated in the
>> > > discussion and voting thread !
>> > >
>> > > Thanks and Regards,
>> > > + Naga
>> > >
>> > > On Mon, Sep 10, 2018 at 2:50 PM Zian Chen 
>> wrote:
>> > >
>> > >> +1 for merge.
>> > >>
>> > >> > On Sep 9, 2018, at 10:47 PM, Weiwei Yang 
>> wrote:
>> > >> >
>> > >> > +1 for the merge
>> > >> >
>> > >> > On Mon, Sep 10, 2018 at 12:06 PM Rohith Sharma K S <
>> > >> > rohithsharm...@apache.org> wrote:
>> > >> >
>> > >> >> +1 for merge
>> > >> >>
>> > >> >> -Rohith Sharma K S
>> > >> >>
>> > >> >> On Wed, 5 Sep 2018 at 18:01, Naganarasimha Garla <
>> > >> >> naganarasimha...@apache.org> wrote:
>> > >> >>
>> > >> >>> Hi All,
>> > >> >>>Thanks for feedback folks, based on the positive response
>> > >> >> starting
>> > >> >>> a Vote thread for merging YARN-3409 to master.
>> > >> >>>
>> > >> >>> Regards,
>> > >> >>> + Naga & Sunil
>> > >> >>>
>> > >> >>> On Wed, 5 Sep 2018 2:51 am Wangda Tan, 
>> wrote:
>> > >> >>>
>> > >>  +1 for the merge, it gonna be a great addition to 3.2.0 release.
>> > >> Thanks
>> > >> >>> to
>> > >>  everybody for pushing this feature to complete.
>> > >> 
>> > >>  Best,
>> > >>  Wangda
>> > >> 
>> > >>  On Tue, Sep 4, 2018 at 8:25 AM Bibinchundatt <
>> > >> >> bibin.chund...@huawei.com>
>> > >>  wrote:
>> > >> 
>> > >> > +1 for merge. Fetaure would be a good addition to 3.2 release.
>> > >> >
>> > >> > --
>> > >> > Bibin A Chundatt
>> > >> > M: +91-9742095715 <+91%2097420%2095715>
>> > >> > E: bibin.chund...@huawei.com
>> > >> > 2012实验室-印研IT&Cloud BU分部
>> > >> > 2012 Laboratories-IT&Cloud BU Branch Dept.
>> > >> > From:Naganarasimha Garla
>> > >> > To:common-dev@hadoop.apache.org,Hdfs-dev,
>> > yarn-...@hadoop.apache.org
>> > >> ,
>> > >> > mapreduce-...@hadoop.apache.org,
>> > >> > Date:2018-08-29 20:00:44
>> > >> > Subject:[Discuss] Merge discussion for Node attribute support
>> > >> feature
>> > >> > YARN-3409
>> > >> >
>> > >> > Hi All,
>> > >> >
>> > >> > We would like to hear your thoughts on merging “Node Attributes
>> > >> >> Support
>> > >> >>> in
>> > >> > YARN” branch (YARN-3409) [2] into trunk in a few weeks. The
>> goal
>> > is
>> > >> to
>> > >> >>> get
>> > >> > it in for HADOOP 3.2.
>> > >> >
>> > >> > *Major work happened in this branch*
>> > >> >
>> > >> > YARN-6858. Attribute Manager to store and provide node
>> attributes
>> > in
>> > >> >> RM
>> > >> > YARN-7871. Support Node attributes reporting from NM to RM(
>> > >> >> distributed
>> > >> > node attributes)
>> > >> > YARN-7863. Modify placement constraints to support node
>> attributes
>> > >> > YARN-7875. Node Attribute store for storing and recovering
>> > >> attributes
>> > >> >
>> > >> > *Detailed Design:*
>> > >> >
>> > >> > Please refer [1] for detailed design document.
>> > >> >
>> > >> > *Testing Efforts:*
>> > >> >
>> > >> > We did detailed tests for the feature in the last few weeks.
>> > >> > This feature will be enabled only when Node Attributes
>> constraints
>> > >> are
>> > >> > specified through SchedulingRequest from AM.
>> > >> > Manager implementation will help to store and recover Node
>> > >> Attributes.
>> > >> > This
>> > >> > works with existing placement constraints.
>> > >> >
>> > >> > *Regarding to API stability:*
>> > >> >
>> > >> > All newly added @Public APIs are @Unstable.
>> > >> >
>> > >> > Documentation jira [3] could help to provide detailed
>> > configuration
>> > >> > details. This feature

Re: [Vote] Merge discussion for Node attribute support feature YARN-3409

2018-09-12 Thread Sunil G

Thanks Sean

YARN-8768 is raised and we will get this in very soon.

- Sunil

On Wed, Sep 12, 2018 at 9:16 PM Sean Mackrory  wrote:

> `mvn install` fails on trunk right now due to javadoc error in yarn-api. Is
> that related? If so, can we revert or fix ASAP please?
>
> On Wed, Sep 12, 2018 at 6:05 AM Naganarasimha Garla <
> naganarasimha...@apache.org> wrote:
>
> > Hi All,
> >  Me and Sunil have successfully merged YARN-3409 to the trunk and
> > we have created new umbrella jira YARN-8766 to track the pending jiras
> and
> > the remaining planned improvement features.
> > Thanks all for the support !
> >
> > Regards,
> > + Naga
> >
> >
> > On Wed, Sep 12, 2018 at 7:46 AM Naganarasimha Garla <
> > naganarasimha...@apache.org> wrote:
> >
> > > Hi All,
> > >  Voting has been running since 6 days and adding my vote we
> have
> > 4
> > > binding and 2 non binding +1's with no -1's this voting passes and we
> > will
> > > be merging the branch shortly. Thanks for all who participated in the
> > > discussion and voting thread !
> > >
> > > Thanks and Regards,
> > > + Naga
> > >
> > > On Mon, Sep 10, 2018 at 2:50 PM Zian Chen 
> wrote:
> > >
> > >> +1 for merge.
> > >>
> > >> > On Sep 9, 2018, at 10:47 PM, Weiwei Yang 
> wrote:
> > >> >
> > >> > +1 for the merge
> > >> >
> > >> > On Mon, Sep 10, 2018 at 12:06 PM Rohith Sharma K S <
> > >> > rohithsharm...@apache.org> wrote:
> > >> >
> > >> >> +1 for merge
> > >> >>
> > >> >> -Rohith Sharma K S
> > >> >>
> > >> >> On Wed, 5 Sep 2018 at 18:01, Naganarasimha Garla <
> > >> >> naganarasimha...@apache.org> wrote:
> > >> >>
> > >> >>> Hi All,
> > >> >>>Thanks for feedback folks, based on the positive response
> > >> >> starting
> > >> >>> a Vote thread for merging YARN-3409 to master.
> > >> >>>
> > >> >>> Regards,
> > >> >>> + Naga & Sunil
> > >> >>>
> > >> >>> On Wed, 5 Sep 2018 2:51 am Wangda Tan, 
> wrote:
> > >> >>>
> > >>  +1 for the merge, it gonna be a great addition to 3.2.0 release.
> > >> Thanks
> > >> >>> to
> > >>  everybody for pushing this feature to complete.
> > >> 
> > >>  Best,
> > >>  Wangda
> > >> 
> > >>  On Tue, Sep 4, 2018 at 8:25 AM Bibinchundatt <
> > >> >> bibin.chund...@huawei.com>
> > >>  wrote:
> > >> 
> > >> > +1 for merge. Fetaure would be a good addition to 3.2 release.
> > >> >
> > >> > --
> > >> > Bibin A Chundatt
> > >> > M: +91-9742095715 <+91%2097420%2095715>
> > >> > E: bibin.chund...@huawei.com
> > >> > 2012实验室-印研IT&Cloud BU分部
> > >> > 2012 Laboratories-IT&Cloud BU Branch Dept.
> > >> > From:Naganarasimha Garla
> > >> > To:common-dev@hadoop.apache.org,Hdfs-dev,
> > yarn-...@hadoop.apache.org
> > >> ,
> > >> > mapreduce-...@hadoop.apache.org,
> > >> > Date:2018-08-29 20:00:44
> > >> > Subject:[Discuss] Merge discussion for Node attribute support
> > >> feature
> > >> > YARN-3409
> > >> >
> > >> > Hi All,
> > >> >
> > >> > We would like to hear your thoughts on merging “Node Attributes
> > >> >> Support
> > >> >>> in
> > >> > YARN” branch (YARN-3409) [2] into trunk in a few weeks. The goal
> > is
> > >> to
> > >> >>> get
> > >> > it in for HADOOP 3.2.
> > >> >
> > >> > *Major work happened in this branch*
> > >> >
> > >> > YARN-6858. Attribute Manager to store and provide node
> attributes
> > in
> > >> >> RM
> > >> > YARN-7871. Support Node attributes reporting from NM to RM(
> > >> >> distributed
> > >> > node attributes)
> > >> > YARN-7863. Modify placement constraints to support node
> attributes
> > >> > YARN-7875. Node Attribute store for storing and recovering
> > >> attributes
> > >> >
> > >> > *Detailed Design:*
> > >> >
> > >> > Please refer [1] for detailed design document.
> > >> >
> > >> > *Testing Efforts:*
> > >> >
> > >> > We did detailed tests for the feature in the last few weeks.
> > >> > This feature will be enabled only when Node Attributes
> constraints
> > >> are
> > >> > specified through SchedulingRequest from AM.
> > >> > Manager implementation will help to store and recover Node
> > >> Attributes.
> > >> > This
> > >> > works with existing placement constraints.
> > >> >
> > >> > *Regarding to API stability:*
> > >> >
> > >> > All newly added @Public APIs are @Unstable.
> > >> >
> > >> > Documentation jira [3] could help to provide detailed
> > configuration
> > >> > details. This feature works from end-to-end and we tested this
> in
> > >> our
> > >> > local
> > >> > cluster. Branch code is run against trunk and tracked via [4].
> > >> >
> > >> > We would love to get your thoughts before opening a voting
> thread.
> > >> >
> > >> > Special thanks to a team of folks who worked hard and
> contributed
> > >> >>> towards
> > >> > this efforts including

Re: [VOTE] Release Apache Hadoop 2.8.5 (RC0)

2018-09-12 Thread Ajay Kumar

Built from source. Not sure if I am missing some config but MR test jobs (Pi) 
are failing with error. ("Error: Could not find or load main class 
org.apache.hadoop.mapreduce.v2.app.MRAppMaster".) 
 Anyone else facing same issue?

See below exception in Nodemanager which I haven't seen earlier.
2018-09-11 11:26:31,343 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: 
NodeManager metrics system started
2018-09-11 11:26:31,386 WARN 
org.apache.hadoop.yarn.util.ResourceCalculatorPlugin: 
java.lang.UnsupportedOperationException: Could not determine OS: Failed to 
instantiate default resource calculator.
java.lang.UnsupportedOperationException: Could not determine OS
at org.apache.hadoop.util.SysInfo.newInstance(SysInfo.java:43)
at 
org.apache.hadoop.yarn.util.ResourceCalculatorPlugin.(ResourceCalculatorPlugin.java:41)
at 
org.apache.hadoop.yarn.util.ResourceCalculatorPlugin.getResourceCalculatorPlugin(ResourceCalculatorPlugin.java:191)
at 
org.apache.hadoop.yarn.server.nodemanager.NodeResourceMonitorImpl.serviceInit(NodeResourceMonitorImpl.java:74)
at 
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at 
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at 
org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:357)
at 
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at 
org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:636)
at 
org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:684)

Thanks,
Ajay

On 9/12/18, 6:31 AM, "Kihwal Lee"  wrote:

+1 (binding)

- Built from source
- Brought up a single node cluster
- Checked basic HDFS functions and checked the UIs
- Ran several simple jobs.

On Mon, Sep 10, 2018 at 7:01 AM 俊平堵  wrote:

> Hi all,
>
>  I've created the first release candidate (RC0) for Apache
> Hadoop 2.8.5. This is our next point release to follow up 2.8.4. It
> includes 33 important fixes and improvements.
>
>
> The RC artifacts are available at:
> http://home.apache.org/~junping_du/hadoop-2.8.5-RC0
>
>
> The RC tag in git is: release-2.8.5-RC0
>
>
>
> The maven artifacts are available via repository.apache.org<
> http://repository.apache.org> at:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1140
>
>
> Please try the release and vote; the vote will run for the usual 5
> working
> days, ending on 9/15/2018 PST time.
>
>
> Thanks,
>
>
> Junping
>

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [Vote] Merge discussion for Node attribute support feature YARN-3409

2018-09-12 Thread Sean Mackrory

`mvn install` fails on trunk right now due to javadoc error in yarn-api. Is
that related? If so, can we revert or fix ASAP please?

On Wed, Sep 12, 2018 at 6:05 AM Naganarasimha Garla <
naganarasimha...@apache.org> wrote:

> Hi All,
>  Me and Sunil have successfully merged YARN-3409 to the trunk and
> we have created new umbrella jira YARN-8766 to track the pending jiras and
> the remaining planned improvement features.
> Thanks all for the support !
>
> Regards,
> + Naga
>
>
> On Wed, Sep 12, 2018 at 7:46 AM Naganarasimha Garla <
> naganarasimha...@apache.org> wrote:
>
> > Hi All,
> >  Voting has been running since 6 days and adding my vote we have
> 4
> > binding and 2 non binding +1's with no -1's this voting passes and we
> will
> > be merging the branch shortly. Thanks for all who participated in the
> > discussion and voting thread !
> >
> > Thanks and Regards,
> > + Naga
> >
> > On Mon, Sep 10, 2018 at 2:50 PM Zian Chen  wrote:
> >
> >> +1 for merge.
> >>
> >> > On Sep 9, 2018, at 10:47 PM, Weiwei Yang  wrote:
> >> >
> >> > +1 for the merge
> >> >
> >> > On Mon, Sep 10, 2018 at 12:06 PM Rohith Sharma K S <
> >> > rohithsharm...@apache.org> wrote:
> >> >
> >> >> +1 for merge
> >> >>
> >> >> -Rohith Sharma K S
> >> >>
> >> >> On Wed, 5 Sep 2018 at 18:01, Naganarasimha Garla <
> >> >> naganarasimha...@apache.org> wrote:
> >> >>
> >> >>> Hi All,
> >> >>>Thanks for feedback folks, based on the positive response
> >> >> starting
> >> >>> a Vote thread for merging YARN-3409 to master.
> >> >>>
> >> >>> Regards,
> >> >>> + Naga & Sunil
> >> >>>
> >> >>> On Wed, 5 Sep 2018 2:51 am Wangda Tan,  wrote:
> >> >>>
> >>  +1 for the merge, it gonna be a great addition to 3.2.0 release.
> >> Thanks
> >> >>> to
> >>  everybody for pushing this feature to complete.
> >> 
> >>  Best,
> >>  Wangda
> >> 
> >>  On Tue, Sep 4, 2018 at 8:25 AM Bibinchundatt <
> >> >> bibin.chund...@huawei.com>
> >>  wrote:
> >> 
> >> > +1 for merge. Fetaure would be a good addition to 3.2 release.
> >> >
> >> > --
> >> > Bibin A Chundatt
> >> > M: +91-9742095715
> >> > E: bibin.chund...@huawei.com
> >> > 2012实验室-印研IT&Cloud BU分部
> >> > 2012 Laboratories-IT&Cloud BU Branch Dept.
> >> > From:Naganarasimha Garla
> >> > To:common-dev@hadoop.apache.org,Hdfs-dev,
> yarn-...@hadoop.apache.org
> >> ,
> >> > mapreduce-...@hadoop.apache.org,
> >> > Date:2018-08-29 20:00:44
> >> > Subject:[Discuss] Merge discussion for Node attribute support
> >> feature
> >> > YARN-3409
> >> >
> >> > Hi All,
> >> >
> >> > We would like to hear your thoughts on merging “Node Attributes
> >> >> Support
> >> >>> in
> >> > YARN” branch (YARN-3409) [2] into trunk in a few weeks. The goal
> is
> >> to
> >> >>> get
> >> > it in for HADOOP 3.2.
> >> >
> >> > *Major work happened in this branch*
> >> >
> >> > YARN-6858. Attribute Manager to store and provide node attributes
> in
> >> >> RM
> >> > YARN-7871. Support Node attributes reporting from NM to RM(
> >> >> distributed
> >> > node attributes)
> >> > YARN-7863. Modify placement constraints to support node attributes
> >> > YARN-7875. Node Attribute store for storing and recovering
> >> attributes
> >> >
> >> > *Detailed Design:*
> >> >
> >> > Please refer [1] for detailed design document.
> >> >
> >> > *Testing Efforts:*
> >> >
> >> > We did detailed tests for the feature in the last few weeks.
> >> > This feature will be enabled only when Node Attributes constraints
> >> are
> >> > specified through SchedulingRequest from AM.
> >> > Manager implementation will help to store and recover Node
> >> Attributes.
> >> > This
> >> > works with existing placement constraints.
> >> >
> >> > *Regarding to API stability:*
> >> >
> >> > All newly added @Public APIs are @Unstable.
> >> >
> >> > Documentation jira [3] could help to provide detailed
> configuration
> >> > details. This feature works from end-to-end and we tested this in
> >> our
> >> > local
> >> > cluster. Branch code is run against trunk and tracked via [4].
> >> >
> >> > We would love to get your thoughts before opening a voting thread.
> >> >
> >> > Special thanks to a team of folks who worked hard and contributed
> >> >>> towards
> >> > this efforts including design discussion / patch / reviews, etc.:
> >> >> Weiwei
> >> > Yang, Bibin Chundatt, Wangda Tan, Vinod Kumar Vavilappali,
> >> >> Konstantinos
> >> > Karanasos, Arun Suresh, Varun Saxena, Devaraj Kavali, Lei Guo,
> Chong
> >> >>> Chen.
> >> >
> >> > [1] :
> >> >
> >> >
> >> >>>
> >> >>
> >>
> https://issues.apache.org/jira/secure/attachment/12937633/Node-Attributes-Requirements-Design-doc_v2.pdf
> >> > [2] : https://issues.a

[jira] [Resolved] (HADOOP-15752) [s3a] testDestroyNoBucket always fails

2018-09-12 Thread Sean Mackrory (JIRA)



 [ 
https://issues.apache.org/jira/browse/HADOOP-15752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory resolved HADOOP-15752.

Resolution: Duplicate

> [s3a] testDestroyNoBucket always fails
> --
>
> Key: HADOOP-15752
> URL: https://issues.apache.org/jira/browse/HADOOP-15752
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/s3
>Reporter: Sean Mackrory
>Priority: Major
>
> ITestS3GuardToolDynamoDB and ITestS3GuardToolLocal both fail the same way:
> {code}
> [INFO] Running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal
> [ERROR] Tests run: 23, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 
> 73.198 s <<< FAILURE! - in 
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal
> [ERROR] 
> testDestroyNoBucket(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal)  
> Time elapsed: 7.048 s  <<< FAILURE!
> java.lang.AssertionError: Expected a java.io.FileNotFoundException to be 
> thrown, but got the result: : 0
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15752) [s3a] testDestroyNoBucket always fails

2018-09-12 Thread Sean Mackrory (JIRA)

Sean Mackrory created HADOOP-15752:
--

 Summary: [s3a] testDestroyNoBucket always fails
 Key: HADOOP-15752
 URL: https://issues.apache.org/jira/browse/HADOOP-15752
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/s3
Reporter: Sean Mackrory


ITestS3GuardToolDynamoDB and ITestS3GuardToolLocal both fail the same way:

{code}
[INFO] Running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal
[ERROR] Tests run: 23, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 73.198 
s <<< FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal
[ERROR] 
testDestroyNoBucket(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal)  
Time elapsed: 7.048 s  <<< FAILURE!
java.lang.AssertionError: Expected a java.io.FileNotFoundException to be 
thrown, but got the result: : 0
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15751) AWS Data read stack trace in S3a putObjectDirect

2018-09-12 Thread Steve Loughran (JIRA)

Steve Loughran created HADOOP-15751:
---

 Summary: AWS Data read stack trace in S3a putObjectDirect
 Key: HADOOP-15751
 URL: https://issues.apache.org/jira/browse/HADOOP-15751
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/s3, test
Affects Versions: 3.3.0
Reporter: Steve Loughran


(transient) failure in a put to AWS london from UK Bristol; 120+ miles away 
(I.e not that far). 

Error "Data read has a different length than the expected:" after the PUT

This is happening in a delayed listing path, so I'd suspect fault injection 
-except this is happening deep in the AWS layer. Maybe its an observed 
inconsistency at that layer, one we aren't yet handling.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15750) remove obsolete S3A test ITestS3ACredentialsInURL

2018-09-12 Thread Steve Loughran (JIRA)

Steve Loughran created HADOOP-15750:
---

 Summary: remove obsolete S3A test ITestS3ACredentialsInURL
 Key: HADOOP-15750
 URL: https://issues.apache.org/jira/browse/HADOOP-15750
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/s3
Affects Versions: 3.3.0
Reporter: Steve Loughran
Assignee: Steve Loughran


now that we've cut user:pass, need to remove the test ITestS3ACredentialsInURL, 
which fails.

I'd missed this because I've got s3guard enabled in test runs, and it 
automatically skips.  oops.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2018-09-12 Thread Apache Jenkins Server

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/894/

[Sep 11, 2018 4:42:34 AM] (brahma) HDFS-13237. [Documentation] RBF: Mount 
points across multiple
[Sep 11, 2018 5:34:23 AM] (xiao) HDFS-13051. Fix dead lock during async editlog 
rolling if edit queue is
[Sep 11, 2018 3:55:34 PM] (aw) HADOOP-14671. Upgrade Apache Yetus to 0.8.0
[Sep 11, 2018 4:16:33 PM] (jlowe) MAPREDUCE-7135. 
TestTaskAttemptContainerRequest should reset
[Sep 11, 2018 5:19:50 PM] (aengineer) HDDS-222. Remove hdfs command line from 
ozone distribution. Contributed
[Sep 11, 2018 5:44:47 PM] (aengineer) HDDS-431. LeaseManager of CommandWatcher 
is not started. Contributed by
[Sep 11, 2018 6:10:50 PM] (jlowe) MAPREDUCE-7136. TestMRAppMetrics should 
shutdown DefaultMetricsSystem
[Sep 11, 2018 10:15:24 PM] (xyao) HDDS-416. Remove currentPosition from 
ChunkInputStreamEntry. Contributed
[Sep 11, 2018 11:47:33 PM] (aengineer) HDDS-424. Consolidate ozone oz 
parameters to use GNU convention.
[Sep 12, 2018 12:00:04 AM] (aengineer) HDDS-432. Replication of closed 
containers is not working. Contributed
[Sep 12, 2018 12:18:42 AM] (liuml07) HADOOP-14833. Remove s3a user:secret 
authentication. Contributed by
[Sep 12, 2018 1:00:57 AM] (brahma) HDFS-13902. Add JMX, conf and stacks menus 
to the datanode page.




-1 overall


The following subsystems voted -1:
asflicense findbugs hadolint pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml
 

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine
 
   Unread field:FSBasedSubmarineStorageImpl.java:[line 39] 
   Found reliance on default encoding in 
org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceJobSubmitter.generateCommandLaunchScript(RunJobParameters,
 TaskType, Component):in 
org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceJobSubmitter.generateCommandLaunchScript(RunJobParameters,
 TaskType, Component): new java.io.FileWriter(File) At 
YarnServiceJobSubmitter.java:[line 195] 
   
org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceJobSubmitter.generateCommandLaunchScript(RunJobParameters,
 TaskType, Component) may fail to clean up java.io.Writer on checked exception 
Obligation to clean up resource created at YarnServiceJobSubmitter.java:to 
clean up java.io.Writer on checked exception Obligation to clean up resource 
created at YarnServiceJobSubmitter.java:[line 195] is not discharged 
   
org.apache.hadoop.yarn.submarine.runtimes.yarnservice.YarnServiceUtils.getComponentArrayJson(String,
 int, String) concatenates strings using + in a loop At 
YarnServiceUtils.java:using + in a loop At YarnServiceUtils.java:[line 72] 

Failed CTEST tests :

   test_test_libhdfs_threaded_hdfs_static 
   test_libhdfs_threaded_hdfspp_test_shim_static 

Failed junit tests :

   hadoop.util.TestReadWriteDiskValidator 
   hadoop.util.TestBasicDiskValidator 
   hadoop.util.TestDiskCheckerWithDiskIo 
   hadoop.hdfs.TestLeaseRecovery2 
   hadoop.hdfs.client.impl.TestBlockReaderLocal 
   hadoop.yarn.sls.appmaster.TestAMSimulator 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/894/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/894/artifact/out/diff-compile-javac-root.txt
  [304K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/894/artifact/out/diff-checkstyle-root.txt
  [17M]

   hadolint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/894/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   pathlen:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/894/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/894/artifact/out/diff-patch-pylint.txt
  [24K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/894/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/894/artifact/out/diff-patch-shelldocs.txt
  [16K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/894/artifact/out/whitespace-eol.txt
  [9.4M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/894/artifact/out/whitespace-tabs.txt
  [1.1M]

   xml:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/894/artifact/out/xml.txt
  [4.0K]

   findbugs:

Re: [VOTE] Release Apache Hadoop 2.8.5 (RC0)

2018-09-12 Thread Kihwal Lee

+1 (binding)

- Built from source
- Brought up a single node cluster
- Checked basic HDFS functions and checked the UIs
- Ran several simple jobs.


On Mon, Sep 10, 2018 at 7:01 AM 俊平堵  wrote:

> Hi all,
>
>  I've created the first release candidate (RC0) for Apache
> Hadoop 2.8.5. This is our next point release to follow up 2.8.4. It
> includes 33 important fixes and improvements.
>
>
> The RC artifacts are available at:
> http://home.apache.org/~junping_du/hadoop-2.8.5-RC0
>
>
> The RC tag in git is: release-2.8.5-RC0
>
>
>
> The maven artifacts are available via repository.apache.org<
> http://repository.apache.org> at:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1140
>
>
> Please try the release and vote; the vote will run for the usual 5
> working
> days, ending on 9/15/2018 PST time.
>
>
> Thanks,
>
>
> Junping
>

Re: [Vote] Merge discussion for Node attribute support feature YARN-3409

2018-09-12 Thread Naganarasimha Garla

Hi All,
 Me and Sunil have successfully merged YARN-3409 to the trunk and
we have created new umbrella jira YARN-8766 to track the pending jiras and
the remaining planned improvement features.
Thanks all for the support !

Regards,
+ Naga


On Wed, Sep 12, 2018 at 7:46 AM Naganarasimha Garla <
naganarasimha...@apache.org> wrote:

> Hi All,
>  Voting has been running since 6 days and adding my vote we have 4
> binding and 2 non binding +1's with no -1's this voting passes and we will
> be merging the branch shortly. Thanks for all who participated in the
> discussion and voting thread !
>
> Thanks and Regards,
> + Naga
>
> On Mon, Sep 10, 2018 at 2:50 PM Zian Chen  wrote:
>
>> +1 for merge.
>>
>> > On Sep 9, 2018, at 10:47 PM, Weiwei Yang  wrote:
>> >
>> > +1 for the merge
>> >
>> > On Mon, Sep 10, 2018 at 12:06 PM Rohith Sharma K S <
>> > rohithsharm...@apache.org> wrote:
>> >
>> >> +1 for merge
>> >>
>> >> -Rohith Sharma K S
>> >>
>> >> On Wed, 5 Sep 2018 at 18:01, Naganarasimha Garla <
>> >> naganarasimha...@apache.org> wrote:
>> >>
>> >>> Hi All,
>> >>>Thanks for feedback folks, based on the positive response
>> >> starting
>> >>> a Vote thread for merging YARN-3409 to master.
>> >>>
>> >>> Regards,
>> >>> + Naga & Sunil
>> >>>
>> >>> On Wed, 5 Sep 2018 2:51 am Wangda Tan,  wrote:
>> >>>
>>  +1 for the merge, it gonna be a great addition to 3.2.0 release.
>> Thanks
>> >>> to
>>  everybody for pushing this feature to complete.
>> 
>>  Best,
>>  Wangda
>> 
>>  On Tue, Sep 4, 2018 at 8:25 AM Bibinchundatt <
>> >> bibin.chund...@huawei.com>
>>  wrote:
>> 
>> > +1 for merge. Fetaure would be a good addition to 3.2 release.
>> >
>> > --
>> > Bibin A Chundatt
>> > M: +91-9742095715
>> > E: bibin.chund...@huawei.com
>> > 2012实验室-印研IT&Cloud BU分部
>> > 2012 Laboratories-IT&Cloud BU Branch Dept.
>> > From:Naganarasimha Garla
>> > To:common-dev@hadoop.apache.org,Hdfs-dev,yarn-...@hadoop.apache.org
>> ,
>> > mapreduce-...@hadoop.apache.org,
>> > Date:2018-08-29 20:00:44
>> > Subject:[Discuss] Merge discussion for Node attribute support
>> feature
>> > YARN-3409
>> >
>> > Hi All,
>> >
>> > We would like to hear your thoughts on merging “Node Attributes
>> >> Support
>> >>> in
>> > YARN” branch (YARN-3409) [2] into trunk in a few weeks. The goal is
>> to
>> >>> get
>> > it in for HADOOP 3.2.
>> >
>> > *Major work happened in this branch*
>> >
>> > YARN-6858. Attribute Manager to store and provide node attributes in
>> >> RM
>> > YARN-7871. Support Node attributes reporting from NM to RM(
>> >> distributed
>> > node attributes)
>> > YARN-7863. Modify placement constraints to support node attributes
>> > YARN-7875. Node Attribute store for storing and recovering
>> attributes
>> >
>> > *Detailed Design:*
>> >
>> > Please refer [1] for detailed design document.
>> >
>> > *Testing Efforts:*
>> >
>> > We did detailed tests for the feature in the last few weeks.
>> > This feature will be enabled only when Node Attributes constraints
>> are
>> > specified through SchedulingRequest from AM.
>> > Manager implementation will help to store and recover Node
>> Attributes.
>> > This
>> > works with existing placement constraints.
>> >
>> > *Regarding to API stability:*
>> >
>> > All newly added @Public APIs are @Unstable.
>> >
>> > Documentation jira [3] could help to provide detailed configuration
>> > details. This feature works from end-to-end and we tested this in
>> our
>> > local
>> > cluster. Branch code is run against trunk and tracked via [4].
>> >
>> > We would love to get your thoughts before opening a voting thread.
>> >
>> > Special thanks to a team of folks who worked hard and contributed
>> >>> towards
>> > this efforts including design discussion / patch / reviews, etc.:
>> >> Weiwei
>> > Yang, Bibin Chundatt, Wangda Tan, Vinod Kumar Vavilappali,
>> >> Konstantinos
>> > Karanasos, Arun Suresh, Varun Saxena, Devaraj Kavali, Lei Guo, Chong
>> >>> Chen.
>> >
>> > [1] :
>> >
>> >
>> >>>
>> >>
>> https://issues.apache.org/jira/secure/attachment/12937633/Node-Attributes-Requirements-Design-doc_v2.pdf
>> > [2] : https://issues.apache.org/jira/browse/YARN-3409
>> > [3] : https://issues.apache.org/jira/browse/YARN-7865
>> > [4] : https://issues.apache.org/jira/browse/YARN-8718
>> >
>> > Thanks,
>> > + Naga & Sunil Govindan
>> >
>> 
>> >>>
>> >>
>> >
>> >
>> > --
>> > Weiwei Yang
>>
>>

Re: Hadoop 3.2 Release Plan proposal

2018-09-12 Thread Sunil G

Hi All,

Inline with the original 3.2 communication proposal dated 17th July 2018, I
would like to provide more updates.

We are approaching previously proposed code freeze date (September 14,
2018). So I would like to cut 3.2 branch on 17th Sept and point existing
trunk to 3.3 if there are no issues.

*Current Release Plan:*
Feature freeze date : all features to merge by September 7, 2018.
Code freeze date : blockers/critical only, no improvements and
blocker/critical bug-fixes September 14, 2018.
Release date: September 28, 2018

If any critical/blocker tickets which are targeted to 3.2.0, we need to
backport to 3.2 post branch cut.

Here's an updated 3.2.0 feature status:

1. Merged & Completed features:

- (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
Initial cut.
- (Uma) HDFS-10285: HDFS Storage Policy Satisfier
- (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
- (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
and CLI.
- (Naga/Sunil) YARN-3409: Node Attributes support in YARN.
- (Inigo) HDFS-12615: Router-based HDFS federation. Improvement works.

2. Features close to finish:

- (Steve) S3Guard Phase III. Close to commit.
- (Steve) S3a phase V. Close to commit.
- (Steve) Support Windows Azure Storage. Close to commit.

3. Tentative/Cancelled features for 3.2:
- (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
Patch in progress.
- (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
done before Aug 2018.
- (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
as more discussions are on-going.

*Summary of 3.2.0 issues status:*
19 Blocker and Critical issues [1] are open, I am following up with owners
to get status on each of them to get in by Code Freeze date.

[1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
BY priority DESC

Thanks,
Sunil



On Thu, Aug 30, 2018 at 9:59 PM Sunil G  wrote:

> Hi All,
>
> Inline with earlier communication dated 17th July 2018, I would like to
> provide some updates.
>
> We are approaching previously proposed code freeze date (Aug 31).
>
> One of the critical feature Node Attributes feature merge discussion/vote
> is ongoing. Also few other Blocker bugs need a bit more time. With regard
> to this, suggesting to push the feature/code freeze for 2 more weeks to
> accommodate these jiras too.
>
> Proposing Updated changes in plan inline with this:
> Feature freeze date : all features to merge by September 7, 2018.
> Code freeze date : blockers/critical only, no improvements and
>  blocker/critical bug-fixes September 14, 2018.
> Release date: September 28, 2018
>
> If any features in branch which are targeted to 3.2.0, please reply to
> this email thread.
>
> *Here's an updated 3.2.0 feature status:*
>
> 1. Merged & Completed features:
>
> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
> Initial cut.
> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
> - (Sunil) YARN-7494: Multi Node scheduling support in Capacity Scheduler.
> - (Chandni/Eric) YARN-7512: Support service upgrade via YARN Service API
> and CLI.
>
> 2. Features close to finish:
>
> - (Naga/Sunil) YARN-3409: Node Attributes support in YARN. Merge/Vote
> Ongoing.
> - (Rohith) YARN-5742: Serve aggregated logs of historical apps from ATSv2.
> Patch in progress.
> - (Virajit) HDFS-12615: Router-based HDFS federation. Improvement works.
> - (Steve) S3Guard Phase III, S3a phase V, Support Windows Azure Storage.
> In progress.
>
> 3. Tentative features:
>
> - (Haibo Chen) YARN-1011: Resource overcommitment. Looks challenging to be
> done before Aug 2018.
> - (Eric) YARN-7129: Application Catalog for YARN applications. Challenging
> as more discussions are on-going.
>
> *Summary of 3.2.0 issues status:*
>
> 26 Blocker and Critical issues [1] are open, I am following up with
> owners to get status on each of them to get in by Code Freeze date.
>
> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker,
> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.2.0 ORDER
> BY priority DESC
>
> Thanks,
> Sunil
>
> On Tue, Aug 14, 2018 at 10:30 PM Sunil G  wrote:
>
>> Hi All,
>>
>> Thanks for the feedbacks. Inline with earlier communication dated 17th
>> July 2018, I would like to provide some updates.
>>
>> We are approaching previously proposed feature freeze date (Aug 21, about
>> 7 days from today).
>> If any features in branch which are targeted to 3.2.0, please reply to
>> this email thread.
>> Steve has mentioned about the s3 features which will come close to Code
>> Freeze Date (Aug 31st).
>>
>> *Here's an updated 3.2.0 feature status:*
>>
>> 1. Merged & Completed features:
>>
>> - (Wangda) YARN-8561: Hadoop Submarine project for DeepLearning workloads
>> Initial cut.
>> - (Uma) HDFS-10285: HDFS Storage Policy Satisfier
>>
>> 2. Fe

[jira] [Created] (HADOOP-15749) Unhandled Log4JLogger explicit reference breaks non-log4j users

2018-09-12 Thread Vitalii Diravka (JIRA)

Vitalii Diravka created HADOOP-15749:


 Summary: Unhandled Log4JLogger explicit reference breaks non-log4j 
users
 Key: HADOOP-15749
 URL: https://issues.apache.org/jira/browse/HADOOP-15749
 Project: Hadoop Common
  Issue Type: Bug
  Components: common
Affects Versions: 3.1.1
Reporter: Vitalii Diravka


The aim of this Jira is the same as HADOOP-10288, but there are more places 
where the issue should be solved, for instance 
{{MetricsLoggerTask.makeMetricsLoggerAsync()}} is called from 
{{NameNode.initialize()}.

Possible solution: the logic from HADOOP-10288 could be factored out to 
{{GenericsUtil}} and can be used where it is necessary.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.5 (RC0)

2018-09-12 Thread Zsolt Venczel

 Thanks for handling the release Junping!

  +1 (non-binding)

I tested the following:
- checked out git tag release-2.8.5-RC0
- built from source on Ubuntu 16.04, java version 8.0.181-oracle
- deployed on a 4 node cluster
- set up a federated cluster with 2 namenodes
- verified teragen, terasort and teravalidate having their output set on
different mountpoints
- enabled snapshots for a mountpoint, created two snapshots, generated a
diff
- checked out the Web UI for the created snapshost that were displaying
correctly

Best regards,
Zsolt

On Wed, Sep 12, 2018 at 12:20 AM Eric Badger 
wrote:

> +1 (non-binding)
>
> - Verified all hashes and checksums
> - Built from source on macOS 10.13.6, Java 1.8.0u65
> - Deployed a pseudo cluster
> - Ran some example jobs
>
> Eric
>
> On Tue, Sep 11, 2018 at 1:39 PM, Gabor Bota  .invalid
> > wrote:
>
> >   Thanks for the work Junping!
> >
> >   +1 (non-binding)
> >
> > - checked out git tag release-2.8.5-RC0
> > - built from source on Mac OS X 10.13.6, java version 8.0.181-oracle
> > - deployed on a 3 node cluster
> > - verified pi job (yarn), teragen, terasort and teravalidate
> >
> >   Regards,
> >   Gabor Bota
> >
> > On Tue, Sep 11, 2018 at 6:31 PM Eric Payne  > invalid>
> > wrote:
> >
> > > Thanks a lot Junping!
> > >
> > > +1 (binding)
> > >
> > > Tested the following:
> > > - Built from source
> > > - Installed on a 7 node, multi-tenant, insecure pseudo cluster, running
> > > YARN capacity scheduler
> > > - Added a queue via refresh
> > > - Verified various GUI pages
> > > - Streaming jobs
> > > - Cross-queue (Inter) preemption
> > > - In-queue (Intra) preemption
> > > - Teragen / terasort
> > >
> > >
> > > -Eric
> > >
> > >
> > >
> > >
> > > On Monday, September 10, 2018, 7:01:46 AM CDT, 俊平堵 <
> > junping...@apache.org>
> > > wrote:
> > >
> > >
> > >
> > >
> > >
> > > Hi all,
> > >
> > > I've created the first release candidate (RC0) for Apache
> > > Hadoop 2.8.5. This is our next point release to follow up 2.8.4. It
> > > includes 33 important fixes and improvements.
> > >
> > >
> > > The RC artifacts are available at:
> > > http://home.apache.org/~junping_du/hadoop-2.8.5-RC0
> > >
> > >
> > > The RC tag in git is: release-2.8.5-RC0
> > >
> > >
> > >
> > > The maven artifacts are available via repository.apache.org<
> > > http://repository.apache.org> at:
> > >
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1140
> > >
> > >
> > > Please try the release and vote; the vote will run for the usual 5
> > > working
> > > days, ending on 9/15/2018 PST time.
> > >
> > >
> > > Thanks,
> > >
> > >
> > > Junping
> > >
> > > -
> > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> > >
> > >
> >
>

[jira] [Created] (HADOOP-15748) S3 listing inconsistency can raise NPE in globber

2018-09-12 Thread Steve Loughran (JIRA)

Steve Loughran created HADOOP-15748:
---

 Summary: S3 listing inconsistency can raise NPE in globber
 Key: HADOOP-15748
 URL: https://issues.apache.org/jira/browse/HADOOP-15748
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs
Affects Versions: 2.8.4, 2.9.1
Reporter: Steve Loughran
Assignee: Steve Loughran


FileSystem Globber does a listStatus(path) and then, if only one element is 
returned, {{getFileStatus(path).isDirectory()}} to see if it is a dir. The way 
getFileStatus() is wrapped, IOEs are downgraded to null

On S3, if the path has had entries deleted, the listing may include files which 
are no longer there, so the getFileStatus(path),isDirectory triggers an NPE

While its wrong to glob against S3 when its being inconsistent, we should at 
least fail gracefully here.

Proposed
# log all IOEs raised in Globber.getFileStatus @ debug
# catch FNFEs and downgrade to warn
# continue

The alternative would be fail fast on FNFE, but that's more traumatic




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15747) warning about user:pass in URI to explicitly call out Hadoop 3.3 as removal

2018-09-12 Thread Steve Loughran (JIRA)

Steve Loughran created HADOOP-15747:
---

 Summary: warning about user:pass in URI to explicitly call out 
Hadoop 3.3 as removal
 Key: HADOOP-15747
 URL: https://issues.apache.org/jira/browse/HADOOP-15747
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/s3
Affects Versions: 3.0.3, 3.1.1, 2.9.1
 Environment: Now that HADOOP-14833 has removed user:secret from URIs, 
change the warning printed when people to that to explicitly declare Hadoop 3.3 
as when it will happen. Do the same in the docs
Reporter: Steve Loughran
Assignee: Steve Loughran






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15754) s3guard: testDynamoTableTagging should clear existing config

[jira] [Created] (HADOOP-15753) ABFS: support path "abfs://mycluster/file/path"

Re: [VOTE] Release Apache Hadoop 2.8.5 (RC0)

Re: [VOTE] Release Apache Hadoop 2.8.5 (RC0)

Re: [Vote] Merge discussion for Node attribute support feature YARN-3409

Re: [Vote] Merge discussion for Node attribute support feature YARN-3409

Re: [Vote] Merge discussion for Node attribute support feature YARN-3409

Re: [VOTE] Release Apache Hadoop 2.8.5 (RC0)

Re: [Vote] Merge discussion for Node attribute support feature YARN-3409

[jira] [Resolved] (HADOOP-15752) [s3a] testDestroyNoBucket always fails

[jira] [Created] (HADOOP-15752) [s3a] testDestroyNoBucket always fails

[jira] [Created] (HADOOP-15751) AWS Data read stack trace in S3a putObjectDirect

[jira] [Created] (HADOOP-15750) remove obsolete S3A test ITestS3ACredentialsInURL

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

Re: [VOTE] Release Apache Hadoop 2.8.5 (RC0)

Re: [Vote] Merge discussion for Node attribute support feature YARN-3409

Re: Hadoop 3.2 Release Plan proposal

[jira] [Created] (HADOOP-15749) Unhandled Log4JLogger explicit reference breaks non-log4j users

Re: [VOTE] Release Apache Hadoop 2.8.5 (RC0)

[jira] [Created] (HADOOP-15748) S3 listing inconsistency can raise NPE in globber

[jira] [Created] (HADOOP-15747) warning about user:pass in URI to explicitly call out Hadoop 3.3 as removal

21 matches

Site Navigation

Mail list logo

Footer information