Build failed in Jenkins: Hadoop-Common-trunk #969

[Apache Jenkins Server]

See 

--
[...truncated 59857 lines...]
Adding reference: maven.local.repository
[DEBUG] Initialize Maven Ant Tasks
parsing buildfile 
jar:file:/home/jenkins/.m2/repository/org/apache/maven/plugins/maven-antrun-plugin/1.7/maven-antrun-plugin-1.7.jar!/org/apache/maven/ant/tasks/antlib.xml
 with URI = 
jar:file:/home/jenkins/.m2/repository/org/apache/maven/plugins/maven-antrun-plugin/1.7/maven-antrun-plugin-1.7.jar!/org/apache/maven/ant/tasks/antlib.xml
 from a zip file
parsing buildfile 
jar:file:/home/jenkins/.m2/repository/org/apache/ant/ant/1.8.2/ant-1.8.2.jar!/org/apache/tools/ant/antlib.xml
 with URI = 
jar:file:/home/jenkins/.m2/repository/org/apache/ant/ant/1.8.2/ant-1.8.2.jar!/org/apache/tools/ant/antlib.xml
 from a zip file
Class org.apache.maven.ant.tasks.AttachArtifactTask loaded from parent loader 
(parentFirst)
 +Datatype attachartifact org.apache.maven.ant.tasks.AttachArtifactTask
Class org.apache.maven.ant.tasks.DependencyFilesetsTask loaded from parent 
loader (parentFirst)
 +Datatype dependencyfilesets org.apache.maven.ant.tasks.DependencyFilesetsTask
Setting project property: test.build.dir -> 

Setting project property: test.exclude.pattern -> _
Setting project property: hadoop.assemblies.version -> 3.0.0-SNAPSHOT
Setting project property: test.exclude -> _
Setting project property: distMgmtSnapshotsId -> apache.snapshots.https
Setting project property: project.build.sourceEncoding -> UTF-8
Setting project property: java.security.egd -> file:///dev/urandom
Setting project property: distMgmtSnapshotsUrl -> 
https://repository.apache.org/content/repositories/snapshots
Setting project property: distMgmtStagingUrl -> 
https://repository.apache.org/service/local/staging/deploy/maven2
Setting project property: avro.version -> 1.7.4
Setting project property: test.build.data -> 

Setting project property: commons-daemon.version -> 1.0.13
Setting project property: hadoop.common.build.dir -> 

Setting project property: testsThreadCount -> 4
Setting project property: maven.test.redirectTestOutputToFile -> true
Setting project property: jdiff.version -> 1.0.9
Setting project property: distMgmtStagingName -> Apache Release Distribution 
Repository
Setting project property: project.reporting.outputEncoding -> UTF-8
Setting project property: build.platform -> Linux-i386-32
Setting project property: protobuf.version -> 2.5.0
Setting project property: failIfNoTests -> false
Setting project property: protoc.path -> ${env.HADOOP_PROTOC_PATH}
Setting project property: jersey.version -> 1.9
Setting project property: distMgmtStagingId -> apache.staging.https
Setting project property: distMgmtSnapshotsName -> Apache Development Snapshot 
Repository
Setting project property: ant.file -> 

[DEBUG] Setting properties with prefix: 
Setting project property: project.groupId -> org.apache.hadoop
Setting project property: project.artifactId -> hadoop-common-project
Setting project property: project.name -> Apache Hadoop Common Project
Setting project property: project.description -> Apache Hadoop Common Project
Setting project property: project.version -> 3.0.0-SNAPSHOT
Setting project property: project.packaging -> pom
Setting project property: project.build.directory -> 

Setting project property: project.build.outputDirectory -> 

Setting project property: project.build.testOutputDirectory -> 

Setting project property: project.build.sourceDirectory -> 

Setting project property: project.build.testSourceDirectory -> 

Setting project property: localRepository ->id: local
  url: file:///home/jenkins/.m2/repository/
   layout: none
Setting project property: settings.localRepository -> 
/home/jenkins/.m2/repository
Setting project property: maven.project.dependencies.versions -> 
[INFO] Executing tasks
Build sequence for target(s) `main' is [main]
Complete build sequence is [main, ]

main:
[mkdir] Created dir: 

[mkdir] Skipping 

Re: Next releases

[Arun C Murthy]

Ok, looks like there are no objections.

I'm starting the work to rename 2.2.1 to 2.3 now. Committers, please hold 
commits till I send out the all clear.

thanks,
Arun

On Nov 20, 2013, at 6:38 AM, Arun C Murthy  wrote:

> Jason,
> 
>  I'm glad to see we are converging. I'll update the Roadmap wiki with details 
> about major/minor/patch releases.
> 
>  Here is a straight-forward approach for now: I'll just roll contents of 
> branch-2.2 as a 2.3-rc0 candidate right-away. This way we don't have to get 
> embroiled in details of individual patches (there are too many). Next up, 
> I'll roll 2.4 in December.
> 
>  Thoughts?
> 
> thanks,
> Arun
> 
> On Nov 13, 2013, at 1:55 PM, Jason Lowe  wrote:
> 
>> I think a lot of confusion comes from the fact that the 2.x line is starting 
>> to mature.  Before this there wasn't such a big contention of what went into 
>> patch vs. minor releases and often the lines were blurred between the two.  
>> However now we have significant customers and products starting to use 2.x 
>> as a base, which means we need to start treating it like we treat 1.x.  That 
>> means getting serious about what we should put into a patch release vs. what 
>> we postpone to a minor release.
>> 
>> Here's my $0.02 on recent proposals:
>> 
>> +1 to releasing more often in general.  A lot of the rush to put changes 
>> into a patch release is because it can be a very long time between any kind 
>> of release.  If minor releases are more frequent then I hope there would be 
>> less of a need to rush something or hold up a release.
>> 
>> +1 to limiting checkins of patch releases to Blockers/Criticals.  If 
>> necessary committers check into trunk/branch-2 only and defer to the patch 
>> release manager for the patch release merge.  Then there should be fewer 
>> surprises for everyone what ended up in a patch release and less likely the 
>> patch release becomes destabilized from the sheer amount of code churn.  
>> Maybe this won't be necessary if everyone understands that the patch release 
>> isn't the only way to get a change out in timely manner.
>> 
>> As for 2.2.1, again I think it's expectations for what that release means.  
>> If it's really just a patch release then there shouldn't be features in it 
>> and tons of code churn, but I think many were treating it as the next 
>> vehicle to deliver changes in general.  If we think 2.2.1 is just as good or 
>> better than 2.2.0 then let's wrap it up and move to a more disciplined 
>> approach for subsequent patch releases and more frequent minor releases.
>> 
>> Jason
>> 
>> On 11/13/2013 12:10 PM, Arun C Murthy wrote:
>>> On Nov 12, 2013, at 1:54 PM, Todd Lipcon  wrote:
>>> 
 On Mon, Nov 11, 2013 at 2:57 PM, Colin McCabe 
 wrote:
 
> To be honest, I'm not aware of anything in 2.2.1 that shouldn't be
> there.  However, I have only been following the HDFS and common side
> of things so I may not have the full picture.  Arun, can you give a
> specific example of something you'd like to "blow away"?
>>> There are bunch of issues in YARN/MapReduce which clearly aren't 
>>> *critical*, similarly in HDFS a cursory glance showed up some 
>>> *enhancements*/*improvements* in CHANGES.txt which aren't necessary for a 
>>> patch release, plus things like:
>>> 
>>> HADOOP-9623 
>>> Update jets3t dependency to 0.9.0
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>>  
>>> Having said that, the HDFS devs know their code the best.
>>> 
 I agree with Colin. If we've been backporting things into a patch release
 (third version component) which don't belong, we should explicitly call out
 those patches, so we can learn from our mistakes and have a discussion
 about what belongs.
>>> Good point.
>>> 
>>> Here is a straw man proposal:
>>> 
>>> 
>>> A patch (third version) release should only include *blocker* bugs which 
>>> are critical from an operational, security or data-integrity issues.
>>> 
>>> This way, we can ensure that a minor series release (2.2.x or 2.3.x or 
>>> 2.4.x) is always release-able, and more importantly, deploy-able at any 
>>> point in time.
>>> 
>>> 
>>> 
>>> Sandy did bring up a related point about timing of releases and the urge 
>>> for everyone to cram features/fixes into a dot release.
>>> 
>>> So, we could remedy that situation by doing a release every 4-6 weeks (2.3, 
>>> 2.4 etc.) and keep the patch releases limited to blocker bugs.
>>> 
>>> Thoughts?
>>> 
>>> thanks,
>>> Arun
>>> 
>>> 
>>> 
>>> 
>>> 
>> 
> 
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
> 
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/



-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, 

Re: Next releases

[Arun C Murthy]

On Dec 2, 2013, at 10:31 AM, Arun C Murthy  wrote:

> Ok, looks like there are no objections.
> 
> I'm starting the work to rename 2.2.1 to 2.3 now. Committers, please hold 
> commits till I send out the all clear.

Done. I've renamed 2.3 -> 2.4 and 2.2.1 -> 2.3.

I'll create the first RC for 2.3 a week from now i.e. 12/9.

thanks,
Arun

> 
> thanks,
> Arun
> 
> On Nov 20, 2013, at 6:38 AM, Arun C Murthy  wrote:
> 
>> Jason,
>> 
>>  I'm glad to see we are converging. I'll update the Roadmap wiki with 
>> details about major/minor/patch releases.
>> 
>>  Here is a straight-forward approach for now: I'll just roll contents of 
>> branch-2.2 as a 2.3-rc0 candidate right-away. This way we don't have to get 
>> embroiled in details of individual patches (there are too many). Next up, 
>> I'll roll 2.4 in December.
>> 
>>  Thoughts?
>> 
>> thanks,
>> Arun
>> 
>> On Nov 13, 2013, at 1:55 PM, Jason Lowe  wrote:
>> 
>>> I think a lot of confusion comes from the fact that the 2.x line is 
>>> starting to mature.  Before this there wasn't such a big contention of what 
>>> went into patch vs. minor releases and often the lines were blurred between 
>>> the two.  However now we have significant customers and products starting 
>>> to use 2.x as a base, which means we need to start treating it like we 
>>> treat 1.x.  That means getting serious about what we should put into a 
>>> patch release vs. what we postpone to a minor release.
>>> 
>>> Here's my $0.02 on recent proposals:
>>> 
>>> +1 to releasing more often in general.  A lot of the rush to put changes 
>>> into a patch release is because it can be a very long time between any kind 
>>> of release.  If minor releases are more frequent then I hope there would be 
>>> less of a need to rush something or hold up a release.
>>> 
>>> +1 to limiting checkins of patch releases to Blockers/Criticals.  If 
>>> necessary committers check into trunk/branch-2 only and defer to the patch 
>>> release manager for the patch release merge.  Then there should be fewer 
>>> surprises for everyone what ended up in a patch release and less likely the 
>>> patch release becomes destabilized from the sheer amount of code churn.  
>>> Maybe this won't be necessary if everyone understands that the patch 
>>> release isn't the only way to get a change out in timely manner.
>>> 
>>> As for 2.2.1, again I think it's expectations for what that release means.  
>>> If it's really just a patch release then there shouldn't be features in it 
>>> and tons of code churn, but I think many were treating it as the next 
>>> vehicle to deliver changes in general.  If we think 2.2.1 is just as good 
>>> or better than 2.2.0 then let's wrap it up and move to a more disciplined 
>>> approach for subsequent patch releases and more frequent minor releases.
>>> 
>>> Jason
>>> 
>>> On 11/13/2013 12:10 PM, Arun C Murthy wrote:
 On Nov 12, 2013, at 1:54 PM, Todd Lipcon  wrote:
 
> On Mon, Nov 11, 2013 at 2:57 PM, Colin McCabe 
> wrote:
> 
>> To be honest, I'm not aware of anything in 2.2.1 that shouldn't be
>> there.  However, I have only been following the HDFS and common side
>> of things so I may not have the full picture.  Arun, can you give a
>> specific example of something you'd like to "blow away"?
 There are bunch of issues in YARN/MapReduce which clearly aren't 
 *critical*, similarly in HDFS a cursory glance showed up some 
 *enhancements*/*improvements* in CHANGES.txt which aren't necessary for a 
 patch release, plus things like:
 
HADOOP-9623 
 Update jets3t dependency to 0.9.0
 
 
 
 
 
 
 
  
 Having said that, the HDFS devs know their code the best.
 
> I agree with Colin. If we've been backporting things into a patch release
> (third version component) which don't belong, we should explicitly call 
> out
> those patches, so we can learn from our mistakes and have a discussion
> about what belongs.
 Good point.
 
 Here is a straw man proposal:
 
 
 A patch (third version) release should only include *blocker* bugs which 
 are critical from an operational, security or data-integrity issues.
 
 This way, we can ensure that a minor series release (2.2.x or 2.3.x or 
 2.4.x) is always release-able, and more importantly, deploy-able at any 
 point in time.
 
 
 
 Sandy did bring up a related point about timing of releases and the urge 
 for everyone to cram features/fixes into a dot release.
 
 So, we could remedy that situation by doing a release every 4-6 weeks 
 (2.3, 2.4 etc.) and keep the patch releases limited to blocker bugs.
 
 Thoughts?
 
 thanks,
 Arun
 
 
 
 
 
>>> 
>> 
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>> 
>> 
> 
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
> 
> 

--
Arun 

[jira] [Created] (HADOOP-10139) Single Cluster Setup document is unfriendly

[Akira AJISAKA (JIRA)]

Akira AJISAKA created HADOOP-10139:
--

 Summary: Single Cluster Setup document is unfriendly
 Key: HADOOP-10139
 URL: https://issues.apache.org/jira/browse/HADOOP-10139
 Project: Hadoop Common
  Issue Type: Bug
  Components: documentation
Affects Versions: 2.2.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA


The document should be understandable to a newcomer because the first place he 
will go is "setup a single node".



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: Next releases

[Arun C Murthy]

On Nov 13, 2013, at 1:55 PM, Jason Lowe  wrote:
> 
> 
> +1 to limiting checkins of patch releases to Blockers/Criticals.  If 
> necessary committers check into trunk/branch-2 only and defer to the patch 
> release manager for the patch release merge.  Then there should be fewer 
> surprises for everyone what ended up in a patch release and less likely the 
> patch release becomes destabilized from the sheer amount of code churn.  
> Maybe this won't be necessary if everyone understands that the patch release 
> isn't the only way to get a change out in timely manner.

I've updated https://wiki.apache.org/hadoop/Roadmap to reflect that we only put 
in Blocker/Critical bugs into Point Releases.

Committers, from now, please exercise extreme caution when committing to a 
point release: they should only be limited to Blocker bugs.

thanks,
Arun


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

branch development for HADOOP-9639

[Sangjin Lee]

We have been having discussions on HADOOP-9639 (shared cache for jars) and
the proposed design there for some time now. We are going to start work on
this and have it vetted and reviewed by the community. I have just filed
some more implementation JIRAs for this feature: YARN-1465, MAPREDUCE-5662,
YARN-1466, YARN-1467

Rather than working privately in our corner and sharing a big patch at the
end, I'd like to explore the idea of developing on a branch in the public
to foster more public feedback. Recently the Hadoop PMC has passed the
change to the bylaws to allow for branch committers (
http://mail-archives.apache.org/mod_mbox/hadoop-general/201307.mbox/%3CCACO5Y4y7HZnn3BS-ZyCVfv-UBcMudeQhndr2vqg%3DXqE1oBiQvQ%40mail.gmail.com%3E),
and I think it would be a good model for this development.

I'd like to propose a branch development and a branch committer status for
a couple of us who are going to work on this per bylaw. Could you please
let me know what you think?

 Thanks,
Sangjin

[jira] [Created] (HADOOP-10140) Specification of HADOOP_CONF_DIR via the environment in hadoop_config.cmd

[Ian Jackson (JIRA)]

Ian Jackson created HADOOP-10140:


 Summary: Specification of HADOOP_CONF_DIR via the environment in 
hadoop_config.cmd
 Key: HADOOP-10140
 URL: https://issues.apache.org/jira/browse/HADOOP-10140
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.2.0
 Environment: This Windows when using the "Command" processing as 
opposed to using Bash from Cygwin.
Reporter: Ian Jackson
Priority: Minor


It would be nice if HADOOP_CONF_DIR could be set in the environment like 
YARN_CONF_DIR. This could be done in lib-exec/hadoop.cmd by setting 
HADOOP_CONF_DIR conditionally.

Using the Windows command shell, the modification would be as follows:
if not defined HADOOP_CONF_DIR (
set HADOOP_CONF_DIR=%HADOOP_HOME%\etc\hadoop
)

This would allow the Hadoop configuration to be placed in ProgramData more 
easily. 




--
This message was sent by Atlassian JIRA
(v6.1#6144)

[VOTE] Merge HDFS-2832 Heterogeneous Storage Phase 1 to trunk

[Arpit Agarwal]

Hello all,

I would like to call a vote to merge phase 1 of the Heterogeneous Storage
feature into trunk.

*Scope of the changes:*
The changes allow exposing the DataNode as a collection of storages and set
the foundation for subsequent work to present Heterogeneous Storages to
applications. This allows DataNodes to send block and storage reports
per-storage. In addition this change introduces the ability to add a
'storage type' tag to the storage directories. This enables supporting
different types of storages in addition to disk storage.

Development of the feature is tracked in the jira
https://issues.apache.org/jira/browse/HDFS-2832.

*Details of development and testing:*
Development has been done in a separate branch -
https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-2832. The
updated design is posted at -
https://issues.apache.org/jira/secure/attachment/12615761/20131125-HeterogeneousStorage.pdf.
The changes involve ~6K changed lines of code, with a third of those
changes being to tests.

Please see the test plan
https://issues.apache.org/jira/secure/attachment/12616642/20131202-HeterogeneousStorage-TestPlan.pdffor
the details. Once the feature is
merged into trunk, we will continue to test and fix any bugs that may be
found on trunk as well as add further tests as outlined in the test plan.

The bulk of the design and implementation was done by Suresh Srinivas,
Sanjay Radia, Nicholas Sze, Junping Du and me. Also, thanks to Eric
Sirianni, Chris Nauroth, Steve Loughran, Bikas Saha, Andrew Wang and Todd
Lipcon for providing feedback on the Jiras and in discussions.

This vote runs for a week and closes on 12/9/2013 at 11:59 pm PT.

Thanks,
Arpit

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Created] (HADOOP-10141) Create an API to separate encryption key storage from applications

[Owen O'Malley (JIRA)]

Owen O'Malley created HADOOP-10141:
--

 Summary: Create an API to separate encryption key storage from 
applications
 Key: HADOOP-10141
 URL: https://issues.apache.org/jira/browse/HADOOP-10141
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Reporter: Owen O'Malley
Assignee: Owen O'Malley


As with the filesystem API, we need to provide a generic mechanism to support 
multiple key storage mechanisms that are potentially from third parties. 

An additional requirement for long term data lakes is to keep multiple versions 
of each key so that keys can be rolled periodically without requiring the 
entire data set to be re-written. Rolling keys provides containment in the 
event of keys being leaked.

Toward that end, I propose an API that is configured using a list of URLs of 
KeyProviders. The implementation will look for implementations using the 
ServiceLoader interface and thus support third party libraries.

Two providers will be included in this patch. One using the credentials cache 
in MapReduce jobs and the other using Java KeyStores from either HDFS or local 
file system. 





--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: [VOTE] Merge HDFS-2832 Heterogeneous Storage Phase 1 to trunk

[Suresh Srinivas]

Great work Arpit and Nicholas!

+1. I have been part of design. I have been following the changes closely.
This is ready to be merged into trunk.




On Mon, Dec 2, 2013 at 4:06 PM, Arpit Agarwal wrote:

> Hello all,
>
> I would like to call a vote to merge phase 1 of the Heterogeneous Storage
> feature into trunk.
>
> *Scope of the changes:*
> The changes allow exposing the DataNode as a collection of storages and set
> the foundation for subsequent work to present Heterogeneous Storages to
> applications. This allows DataNodes to send block and storage reports
> per-storage. In addition this change introduces the ability to add a
> 'storage type' tag to the storage directories. This enables supporting
> different types of storages in addition to disk storage.
>
> Development of the feature is tracked in the jira
> https://issues.apache.org/jira/browse/HDFS-2832.
>
> *Details of development and testing:*
> Development has been done in a separate branch -
> https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-2832. The
> updated design is posted at -
>
> https://issues.apache.org/jira/secure/attachment/12615761/20131125-HeterogeneousStorage.pdf
> .
> The changes involve ~6K changed lines of code, with a third of those
> changes being to tests.
>
> Please see the test plan
>
> https://issues.apache.org/jira/secure/attachment/12616642/20131202-HeterogeneousStorage-TestPlan.pdffor
> the details. Once the feature is
> merged into trunk, we will continue to test and fix any bugs that may be
> found on trunk as well as add further tests as outlined in the test plan.
>
> The bulk of the design and implementation was done by Suresh Srinivas,
> Sanjay Radia, Nicholas Sze, Junping Du and me. Also, thanks to Eric
> Sirianni, Chris Nauroth, Steve Loughran, Bikas Saha, Andrew Wang and Todd
> Lipcon for providing feedback on the Jiras and in discussions.
>
> This vote runs for a week and closes on 12/9/2013 at 11:59 pm PT.
>
> Thanks,
> Arpit
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>



-- 
http://hortonworks.com/download/

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [VOTE] Merge HDFS-2832 Heterogeneous Storage Phase 1 to trunk

[Jun Ping Du]

+1. Good to see HDFS can support different storage tiers. 
I have been involved with minor development & bug fixing effort and I agree it 
is ready to merge too. 

Thanks,

Junping

- Original Message -
From: "Suresh Srinivas" 
To: hdfs-...@hadoop.apache.org
Cc: common-dev@hadoop.apache.org
Sent: Tuesday, December 3, 2013 8:15:26 AM
Subject: Re: [VOTE] Merge HDFS-2832 Heterogeneous Storage Phase 1 to trunk

Great work Arpit and Nicholas!

+1. I have been part of design. I have been following the changes closely.
This is ready to be merged into trunk.




On Mon, Dec 2, 2013 at 4:06 PM, Arpit Agarwal wrote:

> Hello all,
>
> I would like to call a vote to merge phase 1 of the Heterogeneous Storage
> feature into trunk.
>
> *Scope of the changes:*
> The changes allow exposing the DataNode as a collection of storages and set
> the foundation for subsequent work to present Heterogeneous Storages to
> applications. This allows DataNodes to send block and storage reports
> per-storage. In addition this change introduces the ability to add a
> 'storage type' tag to the storage directories. This enables supporting
> different types of storages in addition to disk storage.
>
> Development of the feature is tracked in the jira
> https://urldefense.proofpoint.com/v1/url?u=https://issues.apache.org/jira/browse/HDFS-2832&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=Mw3izENeqbMFnOzHo594vQ%3D%3D%0A&m=Z%2F4r%2B%2FNk9yYikPHTUsPHx9kGN2a1jV0DGMDT3uJYLqw%3D%0A&s=1d962b101464bff32d5f6339d6616e1dc05d7ece8890923c322549257b696ecf.
>
> *Details of development and testing:*
> Development has been done in a separate branch -
> https://urldefense.proofpoint.com/v1/url?u=https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-2832&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=Mw3izENeqbMFnOzHo594vQ%3D%3D%0A&m=Z%2F4r%2B%2FNk9yYikPHTUsPHx9kGN2a1jV0DGMDT3uJYLqw%3D%0A&s=09ee6285bc4abc6bcac8ba4f05f9ef65d66cf1e94d8e705b356bd429659eb689.
>  The
> updated design is posted at -
>
> https://urldefense.proofpoint.com/v1/url?u=https://issues.apache.org/jira/secure/attachment/12615761/20131125-HeterogeneousStorage.pdf&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=Mw3izENeqbMFnOzHo594vQ%3D%3D%0A&m=Z%2F4r%2B%2FNk9yYikPHTUsPHx9kGN2a1jV0DGMDT3uJYLqw%3D%0A&s=9ca29507f2c8e258ab7d4ddcdae82926bf66e4052c31368b2abead87c7df110f
> .
> The changes involve ~6K changed lines of code, with a third of those
> changes being to tests.
>
> Please see the test plan
>
> https://urldefense.proofpoint.com/v1/url?u=https://issues.apache.org/jira/secure/attachment/12616642/20131202-HeterogeneousStorage-TestPlan.pdffor&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=Mw3izENeqbMFnOzHo594vQ%3D%3D%0A&m=Z%2F4r%2B%2FNk9yYikPHTUsPHx9kGN2a1jV0DGMDT3uJYLqw%3D%0A&s=f1d595e4b4cb6ee14cd92093cc1d7f173671f606606528931cd34cab4a08fabd
> the details. Once the feature is
> merged into trunk, we will continue to test and fix any bugs that may be
> found on trunk as well as add further tests as outlined in the test plan.
>
> The bulk of the design and implementation was done by Suresh Srinivas,
> Sanjay Radia, Nicholas Sze, Junping Du and me. Also, thanks to Eric
> Sirianni, Chris Nauroth, Steve Loughran, Bikas Saha, Andrew Wang and Todd
> Lipcon for providing feedback on the Jiras and in discussions.
>
> This vote runs for a week and closes on 12/9/2013 at 11:59 pm PT.
>
> Thanks,
> Arpit
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>



-- 
https://urldefense.proofpoint.com/v1/url?u=http://hortonworks.com/download/&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=Mw3izENeqbMFnOzHo594vQ%3D%3D%0A&m=Z%2F4r%2B%2FNk9yYikPHTUsPHx9kGN2a1jV0DGMDT3uJYLqw%3D%0A&s=167e3a51d20384a08fa0a4a1636e40ce4ecf7a996bfcaccd97e60a3883b42b61

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You

[VOTE] Release Apache Hadoop 0.23.10

[Thomas Graves]

Hey Everyone,

There have been lots of improvements and bug fixes that have went into
branch-0.23 since the 0.23.9 release.  We think its time to do a 0.23.10
so I have created a release candidate (rc0) for a Hadoop-0.23.10 release.

The RC is available at:
http://people.apache.org/~tgraves/hadoop-0.23.10-rc0/


The RC Tag in svn is here:
http://svn.apache.org/viewvc/hadoop/common/tags/release-0.23.10-rc0/


The maven artifacts are available via repository.apache.org.


Please try the release and vote; the vote will run for the usual 7 days
til December 9th.

I am +1 (binding).

thanks,
Tom Graves

[jira] [Created] (HADOOP-10142) Reduce the log generated by ShellBasedUnixGroupsMapping

[Vinay (JIRA)]

Vinay created HADOOP-10142:
--

 Summary: Reduce the log generated by ShellBasedUnixGroupsMapping
 Key: HADOOP-10142
 URL: https://issues.apache.org/jira/browse/HADOOP-10142
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Vinay
Assignee: Vinay


Reduce the logs generated by ShellBasedUnixGroupsMapping.
For ex: Using WebHdfs from windows generates following log for each request

{noformat}2013-12-03 11:34:56,589 WARN 
org.apache.hadoop.security.ShellBasedUnixGroupsMapping: got exception trying to 
get groups for user dr.who
org.apache.hadoop.util.Shell$ExitCodeException: id: dr.who: No such user

at org.apache.hadoop.util.Shell.runCommand(Shell.java:504)
at org.apache.hadoop.util.Shell.run(Shell.java:417)
at 
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:636)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:725)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:708)
at 
org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:83)
at 
org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:52)
at 
org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:50)
at org.apache.hadoop.security.Groups.getGroups(Groups.java:95)
at 
org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1376)
at 
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.(FSPermissionChecker.java:63)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getPermissionChecker(FSNamesystem.java:3228)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListingInt(FSNamesystem.java:4063)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListing(FSNamesystem.java:4052)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getListing(NameNodeRpcServer.java:748)
at 
org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.getDirectoryListing(NamenodeWebHdfsMethods.java:715)
at 
org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.getListingStream(NamenodeWebHdfsMethods.java:727)
at 
org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.get(NamenodeWebHdfsMethods.java:675)
at 
org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.access$400(NamenodeWebHdfsMethods.java:114)
at 
org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$3.run(NamenodeWebHdfsMethods.java:623)
at 
org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods$3.run(NamenodeWebHdfsMethods.java:618)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1515)
at 
org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.get(NamenodeWebHdfsMethods.java:618)
at 
org.apache.hadoop.hdfs.server.namenode.web.resources.NamenodeWebHdfsMethods.getRoot(NamenodeWebHdfsMethods.java:586)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at 
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
at 
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at 
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at 
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at 
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at 
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at 
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at 
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at 
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at 
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebAppli