[jira] [Created] (HDFS-4752) TestRBWBlockInvalidation fails on Windows due to file locking

2013-04-25 Thread Chris Nauroth (JIRA)
Chris Nauroth created HDFS-4752:
---

 Summary: TestRBWBlockInvalidation fails on Windows due to file 
locking
 Key: HDFS-4752
 URL: https://issues.apache.org/jira/browse/HDFS-4752
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: datanode, test
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth


The test attempts to invalidate a block by deleting its block file and meta 
file.  This happens while a datanode thread holds the files open for write.  On 
Windows, this causes a locking conflict, and the test fails.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


Jenkins build became unstable: Hadoop-Hdfs-0.23-Build #592

2013-04-25 Thread Apache Jenkins Server
See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/592/



[jira] [Created] (HDFS-4754) Add an API in the namenode to mark a datanode as stale

2013-04-25 Thread Nicolas Liochon (JIRA)
Nicolas Liochon created HDFS-4754:
-

 Summary: Add an API in the namenode to mark a datanode as stale
 Key: HDFS-4754
 URL: https://issues.apache.org/jira/browse/HDFS-4754
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client, namenode
Reporter: Nicolas Liochon
Priority: Critical


There is a detection of the stale datanodes in HDFS since HDFS-3703, with a 
timeout, defaulted to 30s.

There are two reasons to add an API to mark a node as stale even if the timeout 
is not yet reached:
 1) ZooKeeper can detect that a client is dead at any moment. So, for HBase, we 
sometimes start the recovery before a node is marked staled. (even with 
reasonable settings as: stale: 20s; HBase ZK timeout: 30s
 2) Some third parties could detect that a node is dead before the timeout, 
hence saving us the cost of retrying. An example or such hw is Arista, 
presented here by [~tsuna] 
http://tsunanet.net/~tsuna/fsf-hbase-meetup-april13.pdf, and confirmed in 
HBASE-6290.

As usual, even if the node is dead it can comeback before the 10 minutes limit. 
So I would propose to set a timebound. The API would be

namenode.markStale(String ipAddress, int port, long durationInMs);

After durationInMs, the namenode would again rely only on its heartbeat to 
decide.

Thoughts?

If there is no objections, and if nobody in the hdfs dev team has the time to 
spend some time on it, I will give it a try for branch 2  3.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


Re: Why failed to use Distcp over FTP protocol?

2013-04-25 Thread Daryn Sharp
While I know a lot about the FileSystem design, this is my first foray in 
FTPFileSystem so I don't anything about the assumptions or requirements for the 
ftp server.  The looks like HDFS delete may be returning false when the file 
doesn't exist, whereas FTP is more appropriately throwing an exception.  If 
true, that behavior would explain why ftp isn't working.

It's also hard to tell exactly what's failing because the exception is being 
generated in a finally block, which may be obscuring another exception.

Daryn

On Apr 24, 2013, at 9:37 PM, sam liu wrote:

I could execute:
- hadoop fs -ls ftp://ftpuser:ftpuser@hostname/tmp/testdir
- hadoop fs -lsr ftp://ftpuser:ftpuser@hostname/tmp/testdir

Is there any special requirement to ftp configurations for running distcp tool? 
In my env, if issue 'hadoop fs -lsr ftp://ftpuser:ftpuser@hostname', it will 
return the root path of my linux file system.


2013/4/24 Daryn Sharp da...@yahoo-inc.commailto:da...@yahoo-inc.com
Listing the root is a bit of a special case that is different than N-many 
directories deep.  Can you list 
ftp://hadoopadm:@ftphostname/some/dir/file or 
ftp://hadoopadm:@ftphostname/some/dir?  I suspect ftp fs has a bug, so 
they will fail too.

On Apr 23, 2013, at 8:03 PM, sam liu wrote:

I can success execute hadoop fs -ls 
ftp://hadoopadm:@ftphostnameftp://hadoopadm:@ftphostname/some/path/here,
 it returns the root path of linux system.

But failed to execute hadoop fs -rm 
ftp://hadoopadm:@ftphostname/some/path/here;, and it returns:
rm: Delete failed 
ftp://hadoopadm:ftp://hadoopadm:@ftphostname/some/path/here@ftphostname/some/path/hereftp://hadoopadm:@ftphostname/some/path/here


2013/4/24 Daryn Sharp da...@yahoo-inc.commailto:da...@yahoo-inc.com
The ftp fs is listing the contents of the given path's parent directory, and 
then trying to match the basename of each child path returned against the 
basename of the given path – quite inefficient…  The FNF is it didn't find a 
match for the basename.  It may be that the ftp server isn't returning a 
listing in exactly the expected format so it's being parsed incorrectly.

Does hadoop fs -ls ftp://hadoopadm:@ftphostname/some/path/here; work?  
Or hadoop fs -rm ftp://hadoopadm:@ftphostname/some/path/here;?  Those 
cmds should exercise the same code paths where you are experiencing errors.

Daryn

On Apr 22, 2013, at 9:06 PM, sam liu wrote:

I encountered IOException and FileNotFoundException:

13/04/17 17:11:10 INFO mapred.JobClient: Task Id : attempt_201304160910_2135_m_
00_0, Status : FAILED
java.io.IOException: The temporary job-output directory 
ftp://hadoopadm:@ftphostname/tmp/_distcp_logs_i74spu/_temporary doesn't 
exist!
at 
org.apache.hadoop.mapred.FileOutputCommitter.getWorkPath(FileOutputCommitter.java:250)
at 
org.apache.hadoop.mapred.FileOutputFormat.getTaskOutputPath(FileOutputFormat.java:244)
at 
org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:116)
at 
org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.init(MapTask.java:820)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(AccessController.java:310)
at javax.security.auth.Subject.doAs(Subject.java:573)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1144)
at org.apache.hadoop.mapred.Child.main(Child.java:249)


... ...

13/04/17 17:11:42 INFO mapred.JobClient: Job complete: job_201304160910_2135
13/04/17 17:11:42 INFO mapred.JobClient: Counters: 6
13/04/17 17:11:42 INFO mapred.JobClient:   Job Counters
13/04/17 17:11:42 INFO mapred.JobClient: Failed map tasks=1
13/04/17 17:11:42 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=33785
13/04/17 17:11:42 INFO mapred.JobClient: Launched map tasks=4
13/04/17 17:11:42 INFO mapred.JobClient: Total time spent by all reduces 
waiting after reserving slots (ms)=0
13/04/17 17:11:42 INFO mapred.JobClient: Total time spent by all maps 
waiting after reserving slots (ms)=0
13/04/17 17:11:42 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=6436
13/04/17 17:11:42 INFO mapred.JobClient: Job Failed: # of failed Map Tasks 
exceeded allowed limit. FailedCount: 1. LastFailedTask: 
task_201304160910_2135_m_00
With failures, global counters are inaccurate; consider running with -i
Copy failed: java.io.FileNotFoundException: File 
ftp://hadoopadm:@ftphostname/tmp/_distcp_tmp_i74spu does not exist.
at 
org.apache.hadoop.fs.ftp.FTPFileSystem.getFileStatus(FTPFileSystem.java:419)
at org.apache.hadoop.fs.ftp.FTPFileSystem.delete(FTPFileSystem.java:302)
at org.apache.hadoop.fs.ftp.FTPFileSystem.delete(FTPFileSystem.java:279)
at 

[jira] [Resolved] (HDFS-4749) Use INodeId to identify the corresponding directory node for FSImage saving/loading

2013-04-25 Thread Tsz Wo (Nicholas), SZE (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo (Nicholas), SZE resolved HDFS-4749.
--

   Resolution: Fixed
Fix Version/s: Snapshot (HDFS-2802)

I have committed this.  Thanks, Jing!

 Use INodeId to identify the corresponding directory node for FSImage 
 saving/loading
 ---

 Key: HDFS-4749
 URL: https://issues.apache.org/jira/browse/HDFS-4749
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Jing Zhao
Assignee: Jing Zhao
 Fix For: Snapshot (HDFS-2802)

 Attachments: HDFS-4749.000.patch


 Currently in fsimage, we use the path to locate a directory node for later 
 loading, i.e., when loading a subtree from fsimage, we first read the path of 
 the directory node, and resolve the path to identify the directory node. This 
 brings extra complexity since we need to generate path for directory nodes in 
 both the current tree and snapshot copies.
 As a simplification, we can use INodeId to identify the directory node.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Resolved] (HDFS-4742) Fix appending to a renamed file with snapshot

2013-04-25 Thread Tsz Wo (Nicholas), SZE (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-4742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo (Nicholas), SZE resolved HDFS-4742.
--

   Resolution: Fixed
Fix Version/s: Snapshot (HDFS-2802)

I have committed this.  Thanks, Jing!

 Fix appending to a renamed file with snapshot
 -

 Key: HDFS-4742
 URL: https://issues.apache.org/jira/browse/HDFS-4742
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Jing Zhao
Assignee: Jing Zhao
 Fix For: Snapshot (HDFS-2802)

 Attachments: HDFS-4742.001.patch


 Fix bug for appending a renamed file.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Created] (HDFS-4755) AccessControlException message is changed in snapshot branch

2013-04-25 Thread Tsz Wo (Nicholas), SZE (JIRA)
Tsz Wo (Nicholas), SZE created HDFS-4755:


 Summary: AccessControlException message is changed in snapshot 
branch
 Key: HDFS-4755
 URL: https://issues.apache.org/jira/browse/HDFS-4755
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Tsz Wo (Nicholas), SZE
Priority: Minor


[~rramya] observed the following

- Trunk:
mkdir: org.apache.hadoop.security.AccessControlException: Permission denied: 
user=hrt_qa, access=WRITE, inode=hdfs:hdfs:hdfs:rwx-x-x

- Snapshot branch:
mkdir: org.apache.hadoop.security.AccessControlException: Permission denied: 
user=hrt_qa, access=WRITE, inode=/user/hdfs

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


Re: Class responsible for Registered DataNode.

2013-04-25 Thread Daryn Sharp
DatanodeManager

Daryn

On Apr 25, 2013, at 1:41 AM, Mohammad Mustaqeem wrote:

 Which class keeps track of all the registered DataNodes?
 
 -- 
 *With regards ---*
 *Mohammad Mustaqeem*,
 M.Tech (CSE)
 MNNIT Allahabad
 9026604270



[jira] [Created] (HDFS-4756) Implement ONCRPC and XDR

2013-04-25 Thread Brandon Li (JIRA)
Brandon Li created HDFS-4756:


 Summary: Implement ONCRPC and XDR
 Key: HDFS-4756
 URL: https://issues.apache.org/jira/browse/HDFS-4756
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Brandon Li


This is to track the implementation of ONCRPC(rfc5531) and XDR(rfc4506). 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Created] (HDFS-4758) Disallow nested snapshottable directories

2013-04-25 Thread Tsz Wo (Nicholas), SZE (JIRA)
Tsz Wo (Nicholas), SZE created HDFS-4758:


 Summary: Disallow nested snapshottable directories
 Key: HDFS-4758
 URL: https://issues.apache.org/jira/browse/HDFS-4758
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE


Nested snapshottable directories are supported by the current implementation.  
However, it seems that there are no good use cases for nested snapshottable 
directories.  So we disable it for now until someone has a valid use case for 
it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Reopened] (HDFS-4489) Use InodeID as as an identifier of a file in HDFS protocols and APIs

2013-04-25 Thread Suresh Srinivas (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas reopened HDFS-4489:
---


 Use InodeID as as an identifier of a file in HDFS protocols and APIs
 

 Key: HDFS-4489
 URL: https://issues.apache.org/jira/browse/HDFS-4489
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Reporter: Brandon Li
Assignee: Brandon Li
 Fix For: 2.0.5-beta


 The benefit of using InodeID to uniquely identify a file can be multiple 
 folds. Here are a few of them:
 1. uniquely identify a file cross rename, related JIRAs include HDFS-4258, 
 HDFS-4437.
 2. modification checks in tools like distcp. Since a file could have been 
 replaced or renamed to, the file name and size combination is no t reliable, 
 but the combination of file id and size is unique.
 3. id based protocol support (e.g., NFS)
 4. to make the pluggable block placement policy use fileid instead of 
 filename (HDFS-385).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Reopened] (HDFS-4434) Provide a mapping from INodeId to INode

2013-04-25 Thread Suresh Srinivas (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas reopened HDFS-4434:
---


 Provide a mapping from INodeId to INode
 ---

 Key: HDFS-4434
 URL: https://issues.apache.org/jira/browse/HDFS-4434
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Affects Versions: 3.0.0
Reporter: Brandon Li
Assignee: Suresh Srinivas
 Fix For: 3.0.0, 2.0.5-beta

 Attachments: HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, 
 HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, 
 HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, 
 HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, 
 HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, 
 HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, 
 HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, 
 HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, 
 HDFS-4434.patch


 This JIRA is to provide a way to access the INode via its id. The proposed 
 solution is to have an in-memory mapping from INodeId to INode. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Created] (HDFS-4759) snapshotDiff of two invalid snapshots but with same name returns success

2013-04-25 Thread Ramya Sunil (JIRA)
Ramya Sunil created HDFS-4759:
-

 Summary: snapshotDiff of two invalid snapshots but with same name 
returns success
 Key: HDFS-4759
 URL: https://issues.apache.org/jira/browse/HDFS-4759
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Ramya Sunil
 Fix For: Snapshot (HDFS-2802)


snapshotDiff of two invalid snapshots which have the same names returns a 
success.

$ hadoop dfs -ls /user/foo/hdfs-snapshots/.snapshot
Found 1 items
drwx--   - foo foo  0 2013-04-26 00:53 
/user/foo/hdfs-snapshots/.snapshot/s1

$ hadoop snapshotDiff /user/foo/hdfs-snapshots invalid invalid 
Difference between snapshot invalid and snapshot invalid under directory 
/user/foo/hdfs-snapshots:

-bash-4.1$ echo $?
0


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Resolved] (HDFS-4755) AccessControlException message is changed in snapshot branch

2013-04-25 Thread Tsz Wo (Nicholas), SZE (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-4755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo (Nicholas), SZE resolved HDFS-4755.
--

   Resolution: Fixed
Fix Version/s: Snapshot (HDFS-2802)
 Hadoop Flags: Reviewed

Thanks Jing for reviewing the patch.

I have committed this.

 AccessControlException message is changed in snapshot branch
 

 Key: HDFS-4755
 URL: https://issues.apache.org/jira/browse/HDFS-4755
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Priority: Minor
 Fix For: Snapshot (HDFS-2802)

 Attachments: h4755_20130425.patch


 [~rramya] observed the following
 - Trunk:
 mkdir: org.apache.hadoop.security.AccessControlException: Permission denied: 
 user=hrt_qa, access=WRITE, inode=hdfs:hdfs:hdfs:rwx-x-x
 - Snapshot branch:
 mkdir: org.apache.hadoop.security.AccessControlException: Permission denied: 
 user=hrt_qa, access=WRITE, inode=/user/hdfs

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Created] (HDFS-4760) Update inodeMap after node replacement

2013-04-25 Thread Jing Zhao (JIRA)
Jing Zhao created HDFS-4760:
---

 Summary: Update inodeMap after node replacement
 Key: HDFS-4760
 URL: https://issues.apache.org/jira/browse/HDFS-4760
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Jing Zhao
Assignee: Jing Zhao


Similar with HDFS-4757, we need to update the inodeMap after node replacement. 
Because a lot of node replacement happens in the snapshot branch (e.g., 
INodeDirectory - INodeDirectoryWithSnapshot, INodeDirectory - 
INodeDirectorySnapshottable, INodeFile - INodeFileWithSnapshot ...), this 
becomes a non-trivial issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Resolved] (HDFS-4650) Add rename test in TestSnapshot

2013-04-25 Thread Tsz Wo (Nicholas), SZE (JIRA)

 [ 
https://issues.apache.org/jira/browse/HDFS-4650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsz Wo (Nicholas), SZE resolved HDFS-4650.
--

   Resolution: Fixed
Fix Version/s: Snapshot (HDFS-2802)

I have committed this.  Thanks, Jing!

 Add rename test in TestSnapshot
 ---

 Key: HDFS-4650
 URL: https://issues.apache.org/jira/browse/HDFS-4650
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode, test
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
 Fix For: Snapshot (HDFS-2802)

 Attachments: HDFS-4650.001.patch


 Add more unit tests and update current unit tests to cover different cases 
 for rename with existence of snapshottable directories and snapshots.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


Heads up - 2.0.5-beta

2013-04-25 Thread Arun C Murthy
Gang,

 With hadoop-2.0.4-alpha released, I'd like 2.0.4 to be the final of our 
hadoop-2.x alphas. We have made lots of progress on hadoop-2.x and I believe we 
are nearly there, exciting times!

 As we have discussed previously, I hope to do a final push to stabilize 
hadoop-2.x, release a hadoop-2.0.5-beta in the next month or so; and then 
declare hadoop-2.1 as stable this summer after a short period of intensive 
testing.

 With that in mind, I really want to make a serious push to lock down APIs and 
wire-protocols for hadoop-2.0.5-beta. Thus, we can confidently support 
hadoop-2.x in a compatible manner in the future. So, it's fine to add new 
features, but please ensure that all APIs are frozen for hadoop-2.0.5-beta

 Vinod is helping out on the YARN/MR side and has tagged a number of final 
changes (including some the final API incompatibilities) we'd like to push in 
before we call hadoop-2.x as ready to be supported (Target Version set to 
2.0.5-beta):
 http://s.apache.org/target-hadoop-2.0.5-beta
 Thanks Vinod! (Note some of the sub-tasks of umbrella jiras may not be tagged, 
but their necessity is implied).

 Similarly on HDFS side, can someone please help out by tagging features, 
bug-fixes, protocol/API changes etc.? This way we can ensure HDFS APIs  
protocols are locked down too - I'd really appreciate it!

thanks,
Arun


--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/




Re: Heads up - 2.0.5-beta

2013-04-25 Thread Suresh Srinivas
Thanks for starting this discussion. I volunteer to do a final review of
protocol changes, so we can avoid incompatible changes to API and wire
protocol post 2.0.5 in Common and HDFS.

We have been working really hard on the following features. I would like to
get into 2.x and see it reach HDFS users:
# Snapshots
# NFS gateway for HDFS
# HDFS-347 unix domain socket based short circuits
# Windows support

Other HDFS folks please let me know if I missed anything.

To ensure a timely release of 2.0.5-beta, we should not hold back for
individual features. However, I would like to make necessary API and/or
protocol changes right-away. This will allow us to adding  features in
subsequent releases e.g. hadoop-2.2 or hadoop-2.3 etc without breaking
compatibility. For e.g. even if we don't complete NFS support, making
FileID related changes in 2.0.5-beta will ensure future compatbility.

Regards,
Suresh



On Thu, Apr 25, 2013 at 6:34 PM, Arun C Murthy a...@hortonworks.com wrote:

 Gang,

  With hadoop-2.0.4-alpha released, I'd like 2.0.4 to be the final of our
 hadoop-2.x alphas. We have made lots of progress on hadoop-2.x and I
 believe we are nearly there, exciting times!

  As we have discussed previously, I hope to do a final push to stabilize
 hadoop-2.x, release a hadoop-2.0.5-beta in the next month or so; and then
 declare hadoop-2.1 as stable this summer after a short period of intensive
 testing.

  With that in mind, I really want to make a serious push to lock down APIs
 and wire-protocols for hadoop-2.0.5-beta. Thus, we can confidently support
 hadoop-2.x in a compatible manner in the future. So, it's fine to add new
 features, but please ensure that all APIs are frozen for hadoop-2.0.5-beta

  Vinod is helping out on the YARN/MR side and has tagged a number of final
 changes (including some the final API incompatibilities) we'd like to push
 in before we call hadoop-2.x as ready to be supported (Target Version set
 to 2.0.5-beta):
  http://s.apache.org/target-hadoop-2.0.5-beta
  Thanks Vinod! (Note some of the sub-tasks of umbrella jiras may not be
 tagged, but their necessity is implied).

  Similarly on HDFS side, can someone please help out by tagging features,
 bug-fixes, protocol/API changes etc.? This way we can ensure HDFS APIs 
 protocols are locked down too - I'd really appreciate it!

 thanks,
 Arun


 --
 Arun C. Murthy
 Hortonworks Inc.
 http://hortonworks.com/





-- 
http://hortonworks.com/download/


How to build hadoop-2.0.3-alpha-src project to get a project like hadoop-2.0.3-alpha?

2013-04-25 Thread sam liu
Hi,

I got hadoop-2.0.3-alpha-src.tar.gz and hadoop-2.0.3-alpha.tar.gz, but
found they have different structures as below:
- hadoop-2.0.3-alpha  contains folder/file:
bin  etc  include  lib  libexec  LICENSE.txt  NOTICE.txt  README.txt  sbin
share
- hadoop-2.0.3-alpha-src.tar.gz contains folder/file:
BUILDING.txt   hadoop-client  hadoop-hdfs-project
hadoop-project   hadoop-yarn-project
pom.xml   releasenotes.HDFS.2.0.3-alpha.html
dev-supporthadoop-common-project  hadoop-mapreduce-project
hadoop-project-dist  LICENSE.txt
README.txt
releasenotes.MAPREDUCE.2.0.3-alpha.html
hadoop-assemblies  hadoop-disthadoop-minicluster
hadoop-tools NOTICE.txt
releasenotes.HADOOP.2.0.3-alpha.html  releasenotes.YARN.2.0.3-alpha.html

And then, in hadoop-2.0.3-alpha-src, I successfully run 'mvn package -Pdist
-DskipTests -Dtar', but do not know how to get a target project which has
similar folder/file structure like the downloaded 'hadoop-2.0.3-alpha'
project after building. Any suggestions?

Thanks!

Sam Liu