[jira] [Created] (HDFS-5282) Support built in resolution of /~ to user's home directory: /user/{user.name}

2013-09-30 Thread Kevin Minder (JIRA) 
Kevin Minder created HDFS-5282:
--

 Summary: Support built in resolution of /~ to user's home 
directory: /user/{user.name}
 Key: HDFS-5282
 URL: https://issues.apache.org/jira/browse/HDFS-5282
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Kevin Minder


In many cases it would be very convenient for HDFS (and WebHDFS in particular) 
to support the Unix notion of /~ representing the user's home directory.  This 
would allow for some scripts reusable because they would not contain hard coded 
home directories.




--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: [VOTE] Release Apache Hadoop 2.1.1-beta

2013-09-30 Thread Arun C Murthy 
Done.

On Sep 27, 2013, at 9:07 AM, Robert Kanter  wrote:

> Hi,
> 
> When will the 2.1.1-beta jars be published in maven so downstream projects
> can start using them?  I only see 2.1.0-beta.
> 
> thanks
> - Robert
> 
> 
> On Tue, Sep 24, 2013 at 4:24 PM, Zhijie Shen  wrote:
> 
>> I've added MAPREDUCE-5531 to the blocker list. - Zhijie
>> 
>> 
>> On Tue, Sep 24, 2013 at 3:41 PM, Arun C Murthy 
>> wrote:
>> 
>>> With 4 +1s (3 binding) and no -1s the vote passes. I'll push it out… I'll
>>> make it clear on the release page, that there are some known issues and
>>> that we will follow up very shortly with another release.
>>> 
>>> Meanwhile, let's fix the remaining blockers (please mark them as such
>> with
>>> Target Version 2.1.2-beta).
>>> The current blockers are here:
>>> http://s.apache.org/hadoop-2.1.2-beta-blockers
>>> 
>>> thanks,
>>> Arun
>>> 
>>> On Sep 16, 2013, at 11:38 PM, Arun C Murthy  wrote:
>>> 
 Folks,
 
 I've created a release candidate (rc0) for hadoop-2.1.1-beta that I
>>> would like to get released - this release fixes a number of bugs on top
>> of
>>> hadoop-2.1.0-beta as a result of significant amounts of testing.
 
 If things go well, this might be the last of the *beta* releases of
>>> hadoop-2.x.
 
 The RC is available at:
>>> http://people.apache.org/~acmurthy/hadoop-2.1.1-beta-rc0
 The RC tag in svn is here:
>>> 
>> http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.1-beta-rc0
 
 The maven artifacts are available via repository.apache.org.
 
 Please try the release and vote; the vote will run for the usual 7
>> days.
 
 thanks,
 Arun
 
 
 --
 Arun C. Murthy
 Hortonworks Inc.
 http://hortonworks.com/
 
 
>>> 
>>> --
>>> Arun C. Murthy
>>> Hortonworks Inc.
>>> http://hortonworks.com/
>>> 
>>> 
>>> 
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>> to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified
>> that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender
>> immediately
>>> and delete it from your system. Thank You.
>>> 
>> 
>> 
>> 
>> --
>> Zhijie Shen
>> Hortonworks Inc.
>> http://hortonworks.com/
>> 
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/



-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Created] (HDFS-5281) COMMIT request should not block

2013-09-30 Thread Brandon Li (JIRA) 
Brandon Li created HDFS-5281:


 Summary: COMMIT request should not block
 Key: HDFS-5281
 URL: https://issues.apache.org/jira/browse/HDFS-5281
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: nfs
Reporter: Brandon Li
Assignee: Brandon Li


Currently Commit request is handled synchronously, blocked at most 30 seconds 
before timeout. This JIRA is to make is asynchronous and thus it won't block 
other requests coming from the same channel. 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (HDFS-5280) Corrupted meta files on data nodes prevents DFClient from connecting to data nodes and updating corruption status to name node.

2013-09-30 Thread Jinghui Wang (JIRA) 
Jinghui Wang created HDFS-5280:
--

 Summary: Corrupted meta files on data nodes prevents DFClient from 
connecting to data nodes and updating corruption status to name node.
 Key: HDFS-5280
 URL: https://issues.apache.org/jira/browse/HDFS-5280
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: datanode, hdfs-client
Affects Versions: 2.0.4-alpha, 2.1.0-beta, 1.1.1
 Environment: Red hat enterprise 6.4
Hadoop-2.1.0
Reporter: Jinghui Wang


Meta files being corrupted causes the DFSClient not able to connect to the 
datanodes to access the blocks, so DFSClient never perform a read on the block, 
which is what throws the ChecksumException when file blocks are corrupted and 
report to the namenode to mark the block as corrupt.  Since the client never 
got to that far, thus the file status remain as healthy and so are all the 
blocks.

To replicate the error, put a file onto HDFS.
run hadoop fsck /tmp/bogus.csv -files -blocks -location will get that following 
output.
FSCK started for path /tmp/bogus.csv at 11:33:29
/tmp/bogus.csv 109 bytes, 1 block(s):  OK
0. blk_-4255166695856420554_5292 len=109 repl=3

find the block/meta files for 4255166695856420554 by running 
ssh datanode1.address find /hadoop/ -name "*4255166695856420554*" and it will 
get the following output:
/hadoop/data1/hdfs/current/subdir2/blk_-4255166695856420554
/hadoop/data1/hdfs/current/subdir2/blk_-4255166695856420554_5292.meta

now corrupt the meta file by running 
ssh datanode1.address "sed -i -e '1i 1234567891' 
/hadoop/data1/hdfs/current/subdir2/blk_-4255166695856420554_5292.meta" 

now run hadoop fs -cat /tmp/bogus.csv
will show the stack trace of DFSClient failing to connect to the data node with 
the corrupted meta file.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (HDFS-5279) Guard against NullPointerException in NameNode JSP pages before initialization of FSNamesystem.

2013-09-30 Thread Chris Nauroth (JIRA) 
Chris Nauroth created HDFS-5279:
---

 Summary: Guard against NullPointerException in NameNode JSP pages 
before initialization of FSNamesystem.
 Key: HDFS-5279
 URL: https://issues.apache.org/jira/browse/HDFS-5279
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 2.1.1-beta, 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth


HDFS-4372 added tracking of NameNode startup progress.  As part of that change, 
the NameNode HTTP server now starts before initialization of the 
{{FSNamesystem}}.  There are a few code paths remaining in the JSP pages that 
are at risk of causing {{NullPointerException}} if accessed when the 
{{FSNamesystem}} has not been fully initialized.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (HDFS-5278) Reduce memory consumptions of TestDFSClientRetries

2013-09-30 Thread Haohui Mai (JIRA) 
Haohui Mai created HDFS-5278:


 Summary: Reduce memory consumptions of TestDFSClientRetries
 Key: HDFS-5278
 URL: https://issues.apache.org/jira/browse/HDFS-5278
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Haohui Mai
Assignee: Haohui Mai
Priority: Minor


TestDFSClientRetries::testDFSClientRetriesOnBusyBlocks() spawns about 50 
threads during the execution, each of which takes more than 6m memory.  It 
makes debugging it in eclipse under the default settings difficult since it 
triggers the OutOfMemoryException.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (HDFS-5277) hadoop fs -expunge does not work for federated namespace

2013-09-30 Thread Vrushali C (JIRA) 
Vrushali C created HDFS-5277:


 Summary: hadoop fs -expunge does not work for federated namespace 
 Key: HDFS-5277
 URL: https://issues.apache.org/jira/browse/HDFS-5277
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 2.0.5-alpha
Reporter: Vrushali C


We noticed that hadoop fs -expunge command does not work across federated 
namespace. This seems to look at only /user//.Trash instead of 
traversing all available namespace and expunging from individual namespace.




--
This message was sent by Atlassian JIRA
(v6.1#6144)

Hadoop-Hdfs-0.23-Build - Build # 746 - Still Failing

2013-09-30 Thread Apache Jenkins Server 
See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/746/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 7870 lines...]
[ERROR] location: class com.google.protobuf.InvalidProtocolBufferException
[ERROR] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[3313,27]
 cannot find symbol
[ERROR] symbol  : method 
setUnfinishedMessage(org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.OpWriteBlockProto)
[ERROR] location: class com.google.protobuf.InvalidProtocolBufferException
[ERROR] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[3319,8]
 cannot find symbol
[ERROR] symbol  : method makeExtensionsImmutable()
[ERROR] location: class 
org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.OpWriteBlockProto
[ERROR] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[3330,10]
 cannot find symbol
[ERROR] symbol  : method 
ensureFieldAccessorsInitialized(java.lang.Class,java.lang.Class)
[ERROR] location: class com.google.protobuf.GeneratedMessage.FieldAccessorTable
[ERROR] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[3335,31]
 cannot find symbol
[ERROR] symbol  : class AbstractParser
[ERROR] location: package com.google.protobuf
[ERROR] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[3344,4]
 method does not override or implement a method from a supertype
[ERROR] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[4098,12]
 cannot find symbol
[ERROR] symbol  : method 
ensureFieldAccessorsInitialized(java.lang.Class,java.lang.Class)
[ERROR] location: class com.google.protobuf.GeneratedMessage.FieldAccessorTable
[ERROR] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[4371,104]
 cannot find symbol
[ERROR] symbol  : method getUnfinishedMessage()
[ERROR] location: class com.google.protobuf.InvalidProtocolBufferException
[ERROR] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[5264,8]
 getUnknownFields() in 
org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.OpTransferBlockProto 
cannot override getUnknownFields() in com.google.protobuf.GeneratedMessage; 
overridden method is final
[ERROR] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[5284,19]
 cannot find symbol
[ERROR] symbol  : method 
parseUnknownField(com.google.protobuf.CodedInputStream,com.google.protobuf.UnknownFieldSet.Builder,com.google.protobuf.ExtensionRegistryLite,int)
[ERROR] location: class 
org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.OpTransferBlockProto
[ERROR] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[5314,15]
 cannot find symbol
[ERROR] symbol  : method 
setUnfinishedMessage(org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.OpTransferBlockProto)
[ERROR] location: class com.google.protobuf.InvalidProtocolBufferException
[ERROR] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[5317,27]
 cannot find symbol
[ERROR] symbol  : method 
setUnfinishedMessage(org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.OpTransferBlockProto)
[ERROR] location: class com.google.protobuf.InvalidProtocolBufferException
[ERROR] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-0.23-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/generated-sources/java/org/apache/hadoop/hdfs/protocol/proto/DataTransferProtos.java:[5323,8]
 cannot find symbol
[ERROR] symbol  : method makeExtensionsImmutable()
[ERROR] location: class 
org.apache.hadoop.hdfs

Build failed in Jenkins: Hadoop-Hdfs-0.23-Build #746

2013-09-30 Thread Apache Jenkins Server 
See 

--
[...truncated 7677 lines...]
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 
:[270,37]
 cannot find symbol
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 
:[281,30]
 cannot find symbol
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 
:[10533,37]
 cannot find symbol
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 
:[10544,30]
 cannot find symbol
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 
:[8357,37]
 cannot find symbol
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 
:[8368,30]
 cannot find symbol
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 
:[12641,37]
 cannot find symbol
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 
:[12652,30]
 cannot find symbol
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 
:[9741,37]
 cannot find symbol
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 
:[9752,30]
 cannot find symbol
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 
:[1781,37]
 cannot find symbol
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 
:[1792,30]
 cannot find symbol
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 
:[5338,37]
 cannot find symbol
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 
:[5349,30]
 cannot find symbol
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 
:[6290,37]
 cannot find symbol
[ERROR] symbol  : class Parser
[ERROR] location: package com.google.protobuf
[ERROR] 
:[6301,30]
 cannot find sym