[jira] [Created] (HADOOP-13390) GC pressure of NetworkTopology.isAncestor call can be eliminated

[He Tianyi (JIRA)]

He Tianyi created HADOOP-13390:
--

 Summary: GC pressure of NetworkTopology.isAncestor call can be 
eliminated
 Key: HADOOP-13390
 URL: https://issues.apache.org/jira/browse/HADOOP-13390
 Project: Hadoop Common
  Issue Type: Improvement
  Components: net
Affects Versions: 2.7.0, 2.6.0, 2.8.0
Reporter: He Tianyi
Priority: Minor


{{NetworkTopology.isAncestor}} is called in {{NetworkTopology.getLeaf}}, which 
is excessively used in block placement policies ({{chooseRandom}} for example).

Currently, the implementation calls {{getPath}} twice. And {{getPath}} performs 
string concatenation on the fly. On a busy NameNode, this introduces more GC 
pressure and CPU overhead for block allocation. 
Given that network location and node name does not generally change frequently, 
we can cache path as a properly of {{Node}} and update accordingly while 
network location or node name is changing.
Also, one of these {{getPath}} calls in {{getLeaf}} can be eliminated because 
we are expecting identical result in both calls.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Not being able to add HDFS contributors

[Akira AJISAKA]

Today I hit the same issue.
Changing the browser to Firefox worked well for me.

-Akira

On 7/19/16 18:10, Zheng, Kai wrote:

Thanks Andrew for the work around!! It works great ...

Regards,
Kai

-Original Message-
From: Andrew Wang [mailto:andrew.w...@cloudera.com]
Sent: Wednesday, July 20, 2016 8:10 AM
To: Chris Douglas 
Cc: Zheng, Kai ; common-dev@hadoop.apache.org
Subject: Re: Not being able to add HDFS contributors

What works for me is pasting in the JIRA userID, checking "ignore popups from this page" 
to quash the browser alerts, and then hitting the "update"
button.

What's broken is the username auto-complete, actually saving works fine.

On Tue, Jul 19, 2016 at 5:08 PM, Chris Douglas 
wrote:


I had the same problem. Infra was able to add them, but I kept getting
an error. -C

On Tue, Jul 19, 2016 at 2:29 PM, Zheng, Kai  wrote:

Hi,

I tried many times in the week at different time but just found it's
not

possible to add more HDFS contributors. I can add some Hadoop ones, though.
It becomes an issue because without adding someone and assigning
issues to him first, he won't be able to work on it and upload patches ...


Could anyone help look at this? Thx a lot!

Regards,
Kai



-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org




-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org




-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

RE: Not being able to add HDFS contributors

[Zheng, Kai]

Thanks Andrew for the work around!! It works great ...

Regards,
Kai

-Original Message-
From: Andrew Wang [mailto:andrew.w...@cloudera.com] 
Sent: Wednesday, July 20, 2016 8:10 AM
To: Chris Douglas 
Cc: Zheng, Kai ; common-dev@hadoop.apache.org
Subject: Re: Not being able to add HDFS contributors

What works for me is pasting in the JIRA userID, checking "ignore popups from 
this page" to quash the browser alerts, and then hitting the "update"
button.

What's broken is the username auto-complete, actually saving works fine.

On Tue, Jul 19, 2016 at 5:08 PM, Chris Douglas 
wrote:

> I had the same problem. Infra was able to add them, but I kept getting 
> an error. -C
>
> On Tue, Jul 19, 2016 at 2:29 PM, Zheng, Kai  wrote:
> > Hi,
> >
> > I tried many times in the week at different time but just found it's 
> > not
> possible to add more HDFS contributors. I can add some Hadoop ones, though.
> It becomes an issue because without adding someone and assigning 
> issues to him first, he won't be able to work on it and upload patches ...
> >
> > Could anyone help look at this? Thx a lot!
> >
> > Regards,
> > Kai
> >
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: Not being able to add HDFS contributors

[Andrew Wang]

What works for me is pasting in the JIRA userID, checking "ignore popups
from this page" to quash the browser alerts, and then hitting the "update"
button.

What's broken is the username auto-complete, actually saving works fine.

On Tue, Jul 19, 2016 at 5:08 PM, Chris Douglas 
wrote:

> I had the same problem. Infra was able to add them, but I kept getting
> an error. -C
>
> On Tue, Jul 19, 2016 at 2:29 PM, Zheng, Kai  wrote:
> > Hi,
> >
> > I tried many times in the week at different time but just found it's not
> possible to add more HDFS contributors. I can add some Hadoop ones, though.
> It becomes an issue because without adding someone and assigning issues to
> him first, he won't be able to work on it and upload patches ...
> >
> > Could anyone help look at this? Thx a lot!
> >
> > Regards,
> > Kai
> >
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: Not being able to add HDFS contributors

[Chris Douglas]

I had the same problem. Infra was able to add them, but I kept getting
an error. -C

On Tue, Jul 19, 2016 at 2:29 PM, Zheng, Kai  wrote:
> Hi,
>
> I tried many times in the week at different time but just found it's not 
> possible to add more HDFS contributors. I can add some Hadoop ones, though. 
> It becomes an issue because without adding someone and assigning issues to 
> him first, he won't be able to work on it and upload patches ...
>
> Could anyone help look at this? Thx a lot!
>
> Regards,
> Kai
>

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13389) TestS3ATemporaryCredentials.testSTS error

[Steven K. Wong (JIRA)]

Steven K. Wong created HADOOP-13389:
---

 Summary: TestS3ATemporaryCredentials.testSTS error
 Key: HADOOP-13389
 URL: https://issues.apache.org/jira/browse/HADOOP-13389
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/s3
Reporter: Steven K. Wong


{{org.apache.hadoop.fs.s3a.TestS3ATemporaryCredentials.testSTS}} throws a 403 
AccessDenied when run without any AWS credentials (access key and secret key) 
in the config.

{noformat}
com.amazonaws.AmazonServiceException: Cannot call GetSessionToken with session 
credentials (Service: AWSSecurityTokenService; Status Code: 403; Error Code: 
AccessDenied; Request ID: X)
at 
com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)
at 
com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770)
at 
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
at 
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
at 
com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1106)
at 
com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.getSessionToken(AWSSecurityTokenServiceClient.java:355)
at 
org.apache.hadoop.fs.s3a.TestS3ATemporaryCredentials.testSTS(TestS3ATemporaryCredentials.java:105)
{noformat}

It fails because the InstanceProfileCredentialsProvider in the credentials 
chain (on line 91) is used, but an instance profile always provides a temporary 
credential and GetSessionToken requires a long-term (not temporary) credential.

Suggestion on how to fix this test case?




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache MSDN Offer is Back

[Chris Nauroth]

A few months ago, we learned that the offer for ASF committers to get an MSDN 
license had gone away.  I'm happy to report that as of a few weeks ago, that 
offer is back in place.  For more details, committers can check out 
https://svn.apache.org/repos/private/committers and read 
donated-licenses/msdn.txt.

--Chris Nauroth

Re: [DISCUSS] Official Docker Image at release time

[Roman Shaposhnik]

On Tue, Jul 19, 2016 at 8:12 AM, Sean Busbey  wrote:
> FWIW, there is a "blessed" apache area on docker hub now, and it's
> just an INFRA request to point out the needed Dockerfile in the repo.
>
> PMCs can also request write access to bintray hosting of docker images
> for PMC members.
>
> Info on INFRA-8441, example on INFRA-12019.
>
> A Docker image that starts up a pseudo distributed instance would be
> great for new folks. I don't know if it's worth the investment to
> build images that do more complex multi-instance deployments (though
> there is some fledgling work in HBase for tooling that would do this
> for us).

I am with Sean on this one. To make it truly useful outside of a Quickstart VM
use case the community image has to deal with configuration management
in a much more explicit way. Just sticking java bits into a Docker container
won't really solve much.

Thanks,
Roman.

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Not being able to add HDFS contributors

[Zheng, Kai]

Hi,

I tried many times in the week at different time but just found it's not 
possible to add more HDFS contributors. I can add some Hadoop ones, though. It 
becomes an issue because without adding someone and assigning issues to him 
first, he won't be able to work on it and upload patches ...

Could anyone help look at this? Thx a lot!

Regards,
Kai

[DISCUSS] 2.6.x line releases

[Sangjin Lee]

It's been a while since we had a release on the 2.6.x line. Is it time to
get ready for a 2.6.5 release? Are folks using and relying on releases on
2.6.x? If there is enough interest, I could take that on. Let me know.

I also want to gauge the community's interest in maintaining the 2.6.x
line. How long do we maintain this line? What would be a sensible EOL
policy? Note that as the main code lines start diverging (java version,
features, etc.), the cost of maintaining multiple release lines does go up.
I'd love to hear your thoughts.

Regards,
Sangjin

[jira] [Created] (HADOOP-13388) Clean up TestLocalFileSystemPermission

[Andras Bokor (JIRA)]

Andras Bokor created HADOOP-13388:
-

 Summary: Clean up TestLocalFileSystemPermission
 Key: HADOOP-13388
 URL: https://issues.apache.org/jira/browse/HADOOP-13388
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Reporter: Andras Bokor
Assignee: Andras Bokor
 Fix For: 3.0.0-alpha2


FileSystemContractBaseTest#testMkdirsWithUmask is changing umask under the 
filesystem. RawLocalFileSystem reads the config on startup so it will not react 
if we change the umask.
It blocks [HADOOP-7363|https://issues.apache.org/jira/browse/HADOOP-7363] since 
testMkdirsWithUmask test will never work with RawLocalFileSystem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] Official Docker Image at release time

[Sean Busbey]

FWIW, there is a "blessed" apache area on docker hub now, and it's
just an INFRA request to point out the needed Dockerfile in the repo.

PMCs can also request write access to bintray hosting of docker images
for PMC members.

Info on INFRA-8441, example on INFRA-12019.

A Docker image that starts up a pseudo distributed instance would be
great for new folks. I don't know if it's worth the investment to
build images that do more complex multi-instance deployments (though
there is some fledgling work in HBase for tooling that would do this
for us).

On Tue, Jul 19, 2016 at 2:46 AM, Tsuyoshi Ozawa  wrote:
> Hi developers,
>
> Klaus mentioned the availability of an official docker image of Apache
> Hadoop. Is it time that we start to distribute an official docker
> image at release time?
>
> http://mail-archives.apache.org/mod_mbox/hadoop-user/201607.mbox/%3CSG2PR04MB162977CFE150444FA022510FB6370%40SG2PR04MB1629.apcprd04.prod.outlook.com%3E
>
> Thoughts?
>
> Thanks,
> - Tsuyoshi
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>



-- 
busbey

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13387) users always get told off for using S3 —even when not using it.

[Steve Loughran (JIRA)]

Steve Loughran created HADOOP-13387:
---

 Summary: users always get told off for using S3 —even when not 
using it.
 Key: HADOOP-13387
 URL: https://issues.apache.org/jira/browse/HADOOP-13387
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/s3
Affects Versions: 2.8.0
Reporter: Steve Loughran


The warning telling people not to use s3 appears during filesystem 
initialization -even if you aren't using the FS. This is because it is printed 
during static initialization, and when the FS code loads all available 
filesystems, that static code is inited.

It needs to be moved into the init() code of an instance



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-13386) Upgrade Avro to 1.8.x

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HADOOP-13386.
-
Resolution: Duplicate

> Upgrade Avro to 1.8.x
> -
>
> Key: HADOOP-13386
> URL: https://issues.apache.org/jira/browse/HADOOP-13386
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: build
>Reporter: Ben McCann
>
> Avro 1.8.x makes generated classes serializable which makes them much easier 
> to use with Spark. It would be great to upgrade Avro to 1.8.x



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[DISCUSS] Official Docker Image at release time

[Tsuyoshi Ozawa]

Hi developers,

Klaus mentioned the availability of an official docker image of Apache
Hadoop. Is it time that we start to distribute an official docker
image at release time?

http://mail-archives.apache.org/mod_mbox/hadoop-user/201607.mbox/%3CSG2PR04MB162977CFE150444FA022510FB6370%40SG2PR04MB1629.apcprd04.prod.outlook.com%3E

Thoughts?

Thanks,
- Tsuyoshi

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org