svn commit: r1408290 - in /hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/src/site: apt/CLIMiniCluster.apt.vm apt/ClusterSetup.apt.vm apt/SingleCluster.apt.vm resources/ resources
Author: tomwhite
Date: Mon Nov 12 13:57:19 2012
New Revision: 1408290
URL: http://svn.apache.org/viewvc?rev=1408290view=rev
Log:
Merge -r 1407657:1407658 from trunk to branch-2. Fixes: HADOOP-8860. Split
MapReduce and YARN sections in documentation navigation.
Added:
hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/src/site/apt/CLIMiniCluster.apt.vm
- copied unchanged from r1408264,
hadoop/common/branches/branch-2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CLIMiniCluster.apt.vm
hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/src/site/apt/ClusterSetup.apt.vm
- copied unchanged from r1408264,
hadoop/common/branches/branch-2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm
hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/src/site/apt/SingleCluster.apt.vm
- copied unchanged from r1408264,
hadoop/common/branches/branch-2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/SingleCluster.apt.vm
hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/src/site/resources/
hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/src/site/resources/css/
hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/src/site/resources/css/site.css
(with props)
hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/src/site/site.xml
(with props)
Added:
hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/src/site/resources/css/site.css
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/src/site/resources/css/site.css?rev=1408290view=auto
==
---
hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/src/site/resources/css/site.css
(added)
+++
hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/src/site/resources/css/site.css
Mon Nov 12 13:57:19 2012
@@ -0,0 +1,30 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* (the License); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an AS IS BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+#banner {
+ height: 93px;
+ background: none;
+}
+
+#bannerLeft img {
+ margin-left: 30px;
+ margin-top: 10px;
+}
+
+#bannerRight img {
+ margin: 17px;
+}
+
Propchange:
hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/src/site/resources/css/site.css
--
svn:eol-style = native
Added:
hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/src/site/site.xml
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/src/site/site.xml?rev=1408290view=auto
==
---
hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/src/site/site.xml
(added)
+++
hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/src/site/site.xml
Mon Nov 12 13:57:19 2012
@@ -0,0 +1,28 @@
+!--
+ Licensed under the Apache License, Version 2.0 (the License);
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an AS IS BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+--
+project name=Apache Hadoop ${project.version}
+
+ skin
+groupIdorg.apache.maven.skins/groupId
+artifactIdmaven-stylus-skin/artifactId
+version1.2/version
+ /skin
+
+ body
+links
+ item name=Apache Hadoop href=http://hadoop.apache.org//
+/links
+ /body
+
+/project
Propchange:
hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/src/site/site.xml
--
svn:eol-style = native
svn commit: r1408290 - in /hadoop/common/branches/branch-2/hadoop-project/src/site: apt/index.apt.vm site.xml
Author: tomwhite
Date: Mon Nov 12 13:57:19 2012
New Revision: 1408290
URL: http://svn.apache.org/viewvc?rev=1408290view=rev
Log:
Merge -r 1407657:1407658 from trunk to branch-2. Fixes: HADOOP-8860. Split
MapReduce and YARN sections in documentation navigation.
Modified:
hadoop/common/branches/branch-2/hadoop-project/src/site/apt/index.apt.vm
hadoop/common/branches/branch-2/hadoop-project/src/site/site.xml
Modified:
hadoop/common/branches/branch-2/hadoop-project/src/site/apt/index.apt.vm
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-project/src/site/apt/index.apt.vm?rev=1408290r1=1408289r2=1408290view=diff
==
--- hadoop/common/branches/branch-2/hadoop-project/src/site/apt/index.apt.vm
(original)
+++ hadoop/common/branches/branch-2/hadoop-project/src/site/apt/index.apt.vm
Mon Nov 12 13:57:19 2012
@@ -34,7 +34,7 @@ Apache Hadoop ${project.version}
Namenodes.
More details are available in the
- {{{./hadoop-yarn/hadoop-yarn-site/Federation.html}HDFS Federation}}
+ {{{./hadoop-project-dist/hadoop-hdfs/Federation.html}HDFS Federation}}
document.
* {MapReduce NextGen aka YARN aka MRv2}
@@ -65,9 +65,9 @@ Getting Started
The Hadoop documentation includes the information you need to get started
using
Hadoop. Begin with the
- {{{./hadoop-yarn/hadoop-yarn-site/SingleCluster.html}Single Node Setup}}
which
+ {{{./hadoop-project-dist/hadoop-common/SingleCluster.html}Single Node
Setup}} which
shows you how to set up a single-node Hadoop installation. Then move on to
the
- {{{./hadoop-yarn/hadoop-yarn-site/ClusterSetup.html}Cluster Setup}} to learn
how
+ {{{./hadoop-project-dist/hadoop-common/ClusterSetup.html}Cluster Setup}} to
learn how
to set up a multi-node Hadoop installation.
Modified: hadoop/common/branches/branch-2/hadoop-project/src/site/site.xml
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-project/src/site/site.xml?rev=1408290r1=1408289r2=1408290view=diff
==
--- hadoop/common/branches/branch-2/hadoop-project/src/site/site.xml (original)
+++ hadoop/common/branches/branch-2/hadoop-project/src/site/site.xml Mon Nov 12
13:57:19 2012
@@ -48,33 +48,35 @@
menu name=Common inherit=top
item name=Overview href=index.html/
- item name=Single Node Setup
href=hadoop-yarn/hadoop-yarn-site/SingleCluster.html/
- item name=Cluster Setup
href=hadoop-yarn/hadoop-yarn-site/ClusterSetup.html/
- item name=Hadoop Commands
href=hadoop-project-dist/hadoop-common/commands_manual.html/
-/menu
+ item name=Single Node Setup
href=hadoop-project-dist/hadoop-common/SingleCluster.html/
+ item name=Cluster Setup
href=hadoop-project-dist/hadoop-common/ClusterSetup.html/
+ item name=CLI Mini Cluster
href=hadoop-project-dist/hadoop-common/CLIMiniCluster.html/
+ /menu
menu name=HDFS inherit=top
- item name=High Availability
href=hadoop-yarn/hadoop-yarn-site/HDFSHighAvailability.html/
- item name=Federation
href=hadoop-yarn/hadoop-yarn-site/Federation.html/
- item name=WebHDFS REST API
href=hadoop-yarn/hadoop-yarn-site/WebHDFS.html/
+ item name=High Availability
href=hadoop-project-dist/hadoop-hdfs/HDFSHighAvailability.html/
+ item name=Federation
href=hadoop-project-dist/hadoop-hdfs/Federation.html/
+ item name=WebHDFS REST API
href=hadoop-project-dist/hadoop-hdfs/WebHDFS.html/
item name=HttpFS Gateway href=hadoop-hdfs-httpfs/index.html/
- item name=HDFS User Guide
href=hadoop-project-dist/hadoop-hdfs/hdfs_user_guide.html/
/menu
-menu name=Yarn/MapReduce inherit=top
+menu name=MapReduce inherit=top
+ item name=Encrypted Shuffle
href=hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html/
+/menu
+
+menu name=YARN inherit=top
item name=YARN Architecture
href=hadoop-yarn/hadoop-yarn-site/YARN.html/
- item name=Writing Yarn Applications
href=hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html/
+ item name=Writing YARN Applications
href=hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html/
item name=Capacity Scheduler
href=hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html/
item name=Web Application Proxy
href=hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html/
- item name=Encrypted Shuffle
href=hadoop-yarn/hadoop-yarn-site/EncryptedShuffle.html/
- item name=Yarn Commands
href=hadoop-yarn/hadoop-yarn-site/YarnCommands.html/
+ item name=YARN Commands
href=hadoop-yarn/hadoop-yarn-site/YarnCommands.html/
/menu
-menu name=YARN REST API's inherit=top
+menu name=YARN REST APIs inherit=top
item name=Introduction
href=hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html/
item name=Resource Manager
svn commit: r1408425 - in /hadoop/common/branches/branch-trunk-win/hadoop-assemblies/src/main/resources/assemblies: hadoop-dist.xml hadoop-yarn-dist.xml
Author: suresh Date: Mon Nov 12 19:52:13 2012 New Revision: 1408425 URL: http://svn.apache.org/viewvc?rev=1408425view=rev Log: HADOOP-9005. Merge hadoop cmd line scripts from branch-1-win. Contributed by Ivan Mitic. Modified: hadoop/common/branches/branch-trunk-win/hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml hadoop/common/branches/branch-trunk-win/hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml Modified: hadoop/common/branches/branch-trunk-win/hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-trunk-win/hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml?rev=1408425r1=1408424r2=1408425view=diff == --- hadoop/common/branches/branch-trunk-win/hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml (original) +++ hadoop/common/branches/branch-trunk-win/hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml Mon Nov 12 19:52:13 2012 @@ -26,6 +26,9 @@ outputDirectory/bin/outputDirectory excludes exclude*.sh/exclude +exclude*-config.cmd/exclude +excludestart-*.cmd/exclude +excludestop-*.cmd/exclude /excludes fileMode0755/fileMode /fileSet @@ -38,6 +41,7 @@ outputDirectory/libexec/outputDirectory includes include*-config.sh/include +include*-config.cmd/include /includes fileMode0755/fileMode /fileSet @@ -46,9 +50,13 @@ outputDirectory/sbin/outputDirectory includes include*.sh/include +include*.cmd/include /includes excludes excludehadoop-config.sh/exclude +excludehadoop.cmd/exclude +excludehdfs.cmd/exclude +excludehadoop-config.cmd/exclude /excludes fileMode0755/fileMode /fileSet Modified: hadoop/common/branches/branch-trunk-win/hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-trunk-win/hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml?rev=1408425r1=1408424r2=1408425view=diff == --- hadoop/common/branches/branch-trunk-win/hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml (original) +++ hadoop/common/branches/branch-trunk-win/hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml Mon Nov 12 19:52:13 2012 @@ -33,6 +33,7 @@ outputDirectorybin/outputDirectory includes includeyarn/include +includeyarn.cmd/include /includes fileMode0755/fileMode /fileSet @@ -41,6 +42,7 @@ outputDirectorylibexec/outputDirectory includes includeyarn-config.sh/include +includeyarn-config.cmd/include /includes fileMode0755/fileMode /fileSet @@ -52,6 +54,8 @@ includeyarn-daemons.sh/include includestart-yarn.sh/include includestop-yarn.sh/include +includestart-yarn.cmd/include +includestop-yarn.cmd/include /includes fileMode0755/fileMode /fileSet
svn commit: r1408426 - /hadoop/common/branches/branch-trunk-win/hadoop-common-project/hadoop-common/CHANGES.branch-trunk-win.txt
Author: suresh Date: Mon Nov 12 19:56:40 2012 New Revision: 1408426 URL: http://svn.apache.org/viewvc?rev=1408426view=rev Log: Updating the contributors list for HDOOP-9005. Modified: hadoop/common/branches/branch-trunk-win/hadoop-common-project/hadoop-common/CHANGES.branch-trunk-win.txt Modified: hadoop/common/branches/branch-trunk-win/hadoop-common-project/hadoop-common/CHANGES.branch-trunk-win.txt URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-trunk-win/hadoop-common-project/hadoop-common/CHANGES.branch-trunk-win.txt?rev=1408426r1=1408425r2=1408426view=diff == --- hadoop/common/branches/branch-trunk-win/hadoop-common-project/hadoop-common/CHANGES.branch-trunk-win.txt (original) +++ hadoop/common/branches/branch-trunk-win/hadoop-common-project/hadoop-common/CHANGES.branch-trunk-win.txt Mon Nov 12 19:56:40 2012 @@ -48,5 +48,6 @@ branch-trunk-win changes - unreleased HADOOP-8977. Multiple FsShell test failures on Windows. (Chris Nauroth via suresh) - HADOOP-9005. Merge hadoop cmd line scripts from branch-1-win. - (Ivan Mitic via suresh) + HADOOP-9005. Merge hadoop cmd line scripts from branch-1-win. (David Lao, + Bikas Saha, Lauren Yang, Chuan Liu, Thejas M Nair and Ivan Mitic via suresh) +
svn commit: r1408450 - in /hadoop/common/branches/branch-1: ./ src/hdfs/org/apache/hadoop/hdfs/ src/hdfs/org/apache/hadoop/hdfs/server/datanode/ src/test/org/apache/hadoop/hdfs/
Author: szetszwo
Date: Mon Nov 12 20:44:09 2012
New Revision: 1408450
URL: http://svn.apache.org/viewvc?rev=1408450view=rev
Log:
HDFS-1539. A config option for the datanode to fsycn a block file when block is
completely written. Contributed by dhruba
Modified:
hadoop/common/branches/branch-1/CHANGES.txt
hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/DFSConfigKeys.java
hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/server/datanode/DataNode.java
hadoop/common/branches/branch-1/src/test/org/apache/hadoop/hdfs/TestFileCreation.java
Modified: hadoop/common/branches/branch-1/CHANGES.txt
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/CHANGES.txt?rev=1408450r1=1408449r2=1408450view=diff
==
--- hadoop/common/branches/branch-1/CHANGES.txt (original)
+++ hadoop/common/branches/branch-1/CHANGES.txt Mon Nov 12 20:44:09 2012
@@ -320,6 +320,9 @@ Release 1.1.1 - Unreleased
and the related JIRAs: HDFS-278, HDFS-1840, HDFS-1870, HDFS-1890,
HDFS-2810,
HDFS-3646 and HDFS-2240. (szetszwo)
+HDFS-1539. A config option for the datanode to fsycn a block file
+when block is completely written. (dhruba via szetszwo)
+
BUG FIXES
HADOOP-8878. Uppercase namenode hostname causes hadoop dfs calls with
Modified:
hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/DFSConfigKeys.java
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/DFSConfigKeys.java?rev=1408450r1=1408449r2=1408450view=diff
==
---
hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/DFSConfigKeys.java
(original)
+++
hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/DFSConfigKeys.java
Mon Nov 12 20:44:09 2012
@@ -119,6 +119,8 @@ public class DFSConfigKeys extends Commo
public static final int DFS_LIST_LIMIT_DEFAULT = 1000;
public static final String DFS_DATANODE_USE_DN_HOSTNAME =
dfs.datanode.use.datanode.hostname;
public static final boolean DFS_DATANODE_USE_DN_HOSTNAME_DEFAULT = false;
+ public static final String DFS_DATANODE_SYNCONCLOSE_KEY =
dfs.datanode.synconclose;
+ public static final boolean DFS_DATANODE_SYNCONCLOSE_DEFAULT = false;
//Delegation token related keys
public static final String DFS_NAMENODE_DELEGATION_KEY_UPDATE_INTERVAL_KEY
= dfs.namenode.delegation.key.update-interval;
Modified:
hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java?rev=1408450r1=1408449r2=1408450view=diff
==
---
hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
(original)
+++
hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
Mon Nov 12 20:44:09 2012
@@ -62,6 +62,7 @@ class BlockReceiver implements java.io.C
private DataInputStream in = null; // from where data are read
private DataChecksum checksum; // from where chunks of a block can be read
private OutputStream out = null; // to block file at local disk
+ private OutputStream cout = null; // output stream for cehcksum file
private FileDescriptor outFd;
private DataOutputStream checksumOut = null; // to crc file at local disk
private int bytesPerChecksum;
@@ -115,6 +116,7 @@ class BlockReceiver implements java.io.C
this.finalized = false;
if (streams != null) {
this.out = streams.dataOut;
+this.cout = streams.checksumOut;
if (out instanceof FileOutputStream) {
this.outFd = ((FileOutputStream) out).getFD();
} else {
@@ -122,7 +124,7 @@ class BlockReceiver implements java.io.C
+ out.getClass());
}
this.checksumOut = new DataOutputStream(new BufferedOutputStream(
- streams.checksumOut,
+ streams.checksumOut,
SMALL_BUFFER_SIZE));
// If this block is for appends, then remove it from periodic
// validation.
@@ -160,6 +162,9 @@ class BlockReceiver implements java.io.C
try {
if (checksumOut != null) {
checksumOut.flush();
+if (datanode.syncOnClose (cout instanceof FileOutputStream)) {
+ ((FileOutputStream)cout).getChannel().force(true);
+}
checksumOut.close();
checksumOut = null;
}
@@ -170,6 +175,9 @@ class BlockReceiver implements java.io.C
try {
if
svn commit: r1408452 - in /hadoop/common/branches/branch-1.1: ./ src/hdfs/org/apache/hadoop/hdfs/ src/hdfs/org/apache/hadoop/hdfs/server/datanode/ src/test/org/apache/hadoop/hdfs/
Author: szetszwo
Date: Mon Nov 12 20:45:33 2012
New Revision: 1408452
URL: http://svn.apache.org/viewvc?rev=1408452view=rev
Log:
svn merge -c 1408450 from branch-1 for HDFS-1539. A config option for the
datanode to fsycn a block file when block is completely written.
Modified:
hadoop/common/branches/branch-1.1/ (props changed)
hadoop/common/branches/branch-1.1/CHANGES.txt (contents, props changed)
hadoop/common/branches/branch-1.1/src/hdfs/org/apache/hadoop/hdfs/DFSConfigKeys.java
hadoop/common/branches/branch-1.1/src/hdfs/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
hadoop/common/branches/branch-1.1/src/hdfs/org/apache/hadoop/hdfs/server/datanode/DataNode.java
hadoop/common/branches/branch-1.1/src/test/org/apache/hadoop/hdfs/TestFileCreation.java
Propchange: hadoop/common/branches/branch-1.1/
--
Merged /hadoop/common/branches/branch-1:r1408450
Modified: hadoop/common/branches/branch-1.1/CHANGES.txt
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-1.1/CHANGES.txt?rev=1408452r1=1408451r2=1408452view=diff
==
--- hadoop/common/branches/branch-1.1/CHANGES.txt (original)
+++ hadoop/common/branches/branch-1.1/CHANGES.txt Mon Nov 12 20:45:33 2012
@@ -17,6 +17,9 @@ Release 1.1.1 - Unreleased
and the related JIRAs: HDFS-278, HDFS-1840, HDFS-1870, HDFS-1890,
HDFS-2810,
HDFS-3646 and HDFS-2240. (szetszwo)
+HDFS-1539. A config option for the datanode to fsycn a block file
+when block is completely written. (dhruba via szetszwo)
+
BUG FIXES
HADOOP-8878. Uppercase namenode hostname causes hadoop dfs calls with
Propchange: hadoop/common/branches/branch-1.1/CHANGES.txt
--
Merged /hadoop/common/branches/branch-1/CHANGES.txt:r1408450
Modified:
hadoop/common/branches/branch-1.1/src/hdfs/org/apache/hadoop/hdfs/DFSConfigKeys.java
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-1.1/src/hdfs/org/apache/hadoop/hdfs/DFSConfigKeys.java?rev=1408452r1=1408451r2=1408452view=diff
==
---
hadoop/common/branches/branch-1.1/src/hdfs/org/apache/hadoop/hdfs/DFSConfigKeys.java
(original)
+++
hadoop/common/branches/branch-1.1/src/hdfs/org/apache/hadoop/hdfs/DFSConfigKeys.java
Mon Nov 12 20:45:33 2012
@@ -113,6 +113,8 @@ public class DFSConfigKeys extends Commo
public static final int DFS_LIST_LIMIT_DEFAULT = 1000;
public static final String DFS_DATANODE_USE_DN_HOSTNAME =
dfs.datanode.use.datanode.hostname;
public static final boolean DFS_DATANODE_USE_DN_HOSTNAME_DEFAULT = false;
+ public static final String DFS_DATANODE_SYNCONCLOSE_KEY =
dfs.datanode.synconclose;
+ public static final boolean DFS_DATANODE_SYNCONCLOSE_DEFAULT = false;
//Delegation token related keys
public static final String DFS_NAMENODE_DELEGATION_KEY_UPDATE_INTERVAL_KEY
= dfs.namenode.delegation.key.update-interval;
Modified:
hadoop/common/branches/branch-1.1/src/hdfs/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-1.1/src/hdfs/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java?rev=1408452r1=1408451r2=1408452view=diff
==
---
hadoop/common/branches/branch-1.1/src/hdfs/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
(original)
+++
hadoop/common/branches/branch-1.1/src/hdfs/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
Mon Nov 12 20:45:33 2012
@@ -61,6 +61,7 @@ class BlockReceiver implements java.io.C
private DataInputStream in = null; // from where data are read
private DataChecksum checksum; // from where chunks of a block can be read
private OutputStream out = null; // to block file at local disk
+ private OutputStream cout = null; // output stream for cehcksum file
private FileDescriptor outFd;
private DataOutputStream checksumOut = null; // to crc file at local disk
private int bytesPerChecksum;
@@ -114,6 +115,7 @@ class BlockReceiver implements java.io.C
this.finalized = false;
if (streams != null) {
this.out = streams.dataOut;
+this.cout = streams.checksumOut;
if (out instanceof FileOutputStream) {
this.outFd = ((FileOutputStream) out).getFD();
} else {
@@ -121,7 +123,7 @@ class BlockReceiver implements java.io.C
+ out.getClass());
}
this.checksumOut = new DataOutputStream(new BufferedOutputStream(
- streams.checksumOut,
+ streams.checksumOut,
SMALL_BUFFER_SIZE));
// If
[Hadoop Wiki] Update of MountableHDFS by BrockNoland
Dear Wiki user, You have subscribed to a wiki page or wiki category on Hadoop Wiki for change notification. The MountableHDFS page has been changed by BrockNoland: http://wiki.apache.org/hadoop/MountableHDFS?action=diffrev1=16rev2=17 * hdfs-fuse - a google code project is very similar to contrib/fuse-dfs * webdav - hdfs exposed as a webdav resource * mapR - contains a closed source hdfs compatible file system that supports read/write NFS access - * [[https://github.com/brockn/hdfs-nfs-proxy|HDFS NFS Proxy]] - exports HDFS as NFS without use of fuse + * [[https://github.com/brockn/hdfs-nfs-proxy|HDFS NFS Proxy]] - exports HDFS as NFS without use of fuse. Supports Kerberos and re-orders writes so they are written to hdfs sequentially. == Supported Operating Systems ==
svn commit: r1408463 - in /hadoop/common/branches/branch-trunk-win: hadoop-dist/pom.xml hadoop-project-dist/pom.xml
Author: suresh
Date: Mon Nov 12 21:17:09 2012
New Revision: 1408463
URL: http://svn.apache.org/viewvc?rev=1408463view=rev
Log:
HADOOP-9008. Building hadoop tarball fails on Windows. Contributed by Chris
Nauroth.
Modified:
hadoop/common/branches/branch-trunk-win/hadoop-dist/pom.xml
hadoop/common/branches/branch-trunk-win/hadoop-project-dist/pom.xml
Modified: hadoop/common/branches/branch-trunk-win/hadoop-dist/pom.xml
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-trunk-win/hadoop-dist/pom.xml?rev=1408463r1=1408462r2=1408463view=diff
==
--- hadoop/common/branches/branch-trunk-win/hadoop-dist/pom.xml (original)
+++ hadoop/common/branches/branch-trunk-win/hadoop-dist/pom.xml Mon Nov 12
21:17:09 2012
@@ -90,50 +90,6 @@
artifactIdmaven-antrun-plugin/artifactId
executions
execution
-iddist/id
-phaseprepare-package/phase
-goals
- goalrun/goal
-/goals
-configuration
- target
-echo
file=${project.build.directory}/dist-layout-stitching.sh
- run() {
-echo \$ ${@}
-${@}
-res=$?
-if [ $res != 0 ]; then
- echo
- echo Failed!
- echo
- exit $res
-fi
- }
-
- ROOT=`cd ${basedir}/..;pwd`
- echo
- echo Current directory `pwd`
- echo
- run rm -rf hadoop-${project.version}
- run mkdir hadoop-${project.version}
- run cd hadoop-${project.version}
- run cp -r
$ROOT/hadoop-common-project/hadoop-common/target/hadoop-common-${project.version}/*
.
- run cp -r
$ROOT/hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-${project.version}/* .
- run cp -r
$ROOT/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/hadoop-hdfs-httpfs-${project.version}/*
.
- run cp -r
$ROOT/hadoop-yarn-project/target/hadoop-yarn-project-${project.version}/* .
- run cp -r
$ROOT/hadoop-mapreduce-project/target/hadoop-mapreduce-${project.version}/* .
- run cp -r
$ROOT/hadoop-tools/hadoop-tools-dist/target/hadoop-tools-dist-${project.version}/*
.
- echo
- echo Hadoop dist layout available at:
${project.build.directory}/hadoop-${project.version}
- echo
-/echo
-exec executable=sh dir=${project.build.directory}
failonerror=true
- arg line=./dist-layout-stitching.sh/
-/exec
- /target
-/configuration
- /execution
- execution
idtar/id
phasepackage/phase
goals
@@ -141,26 +97,61 @@
/goals
configuration
target if=tar
-echo
file=${project.build.directory}/dist-tar-stitching.sh
- run() {
-echo \$ ${@}
-${@}
-res=$?
-if [ $res != 0 ]; then
- echo
- echo Failed!
- echo
- exit $res
-fi
- }
+!-- This script preserves permissions and symlinks. --
+!-- Python requires resetting indentation to far left. --
+echo file=${project.build.directory}/dist-maketar.py
+from os.path import abspath, basename, isdir, join
+import tarfile
+
+def make_file_filter(root, file_name_filter):
+ def filter_func(tar_info):
+if tar_info.name == root:
+ # Always include root directory. Otherwise, tarfile.add assumes you are
+ # filtering out the whole directory and produces an empty tar.
+ return tar_info
+if tar_info.isfile() or tar_info.issym():
+ # Include files and symlinks only if they match the specified name
filter.
+ if file_name_filter(basename(tar_info.name)):
+return tar_info
+# Otherwise, exclude.
+return None
+ return filter_func
+
+target_dirs = [
+
abspath(r${basedir}/../hadoop-common-project/hadoop-common/target/hadoop-common-${project.version}),
+
abspath(r${basedir}/../hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-${project.version}),
+
abspath(r${basedir}/../hadoop-hdfs-project/hadoop-hdfs-httpfs/target/hadoop-hdfs-httpfs-${project.version}),
+
svn commit: r1408468 - in /hadoop/common/branches/branch-1.1: CHANGES.txt ivy/hadoop-client-pom-template.xml ivy/hadoop-minicluster-pom-template.xml
Author: gkesavan
Date: Mon Nov 12 21:44:06 2012
New Revision: 1408468
URL: http://svn.apache.org/viewvc?rev=1408468view=rev
Log:
HADOOP-9017. Fix hadoop-client-pom-template.xml and
hadoop-client-pom-template.xml for version token. Contributed by Giridharan
Kesavan
Modified:
hadoop/common/branches/branch-1.1/CHANGES.txt
hadoop/common/branches/branch-1.1/ivy/hadoop-client-pom-template.xml
hadoop/common/branches/branch-1.1/ivy/hadoop-minicluster-pom-template.xml
Modified: hadoop/common/branches/branch-1.1/CHANGES.txt
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-1.1/CHANGES.txt?rev=1408468r1=1408467r2=1408468view=diff
==
--- hadoop/common/branches/branch-1.1/CHANGES.txt (original)
+++ hadoop/common/branches/branch-1.1/CHANGES.txt Mon Nov 12 21:44:06 2012
@@ -45,6 +45,10 @@ Release 1.1.1 - Unreleased
MAPREDUCE-4749. Fixed a bug in TaskTracker because of which kill-actions
get
delayed progressively. (Arpit Gupta via vinodkv)
+HADOOP-9017. Fix hadoop-client-pom-template.xml and
+hadoop-client-pom-template.xml for version token ( gkesavan )
+
+
Release 1.1.0 - 2012.09.28
INCOMPATIBLE CHANGES
Modified: hadoop/common/branches/branch-1.1/ivy/hadoop-client-pom-template.xml
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-1.1/ivy/hadoop-client-pom-template.xml?rev=1408468r1=1408467r2=1408468view=diff
==
--- hadoop/common/branches/branch-1.1/ivy/hadoop-client-pom-template.xml
(original)
+++ hadoop/common/branches/branch-1.1/ivy/hadoop-client-pom-template.xml Mon
Nov 12 21:44:06 2012
@@ -32,7 +32,7 @@
dependency
groupIdorg.apache.hadoop/groupId
artifactIdhadoop-core/artifactId
- version${project.version}/version
+ version@version/version
scopecompile/scope
exclusions
exclusion
Modified:
hadoop/common/branches/branch-1.1/ivy/hadoop-minicluster-pom-template.xml
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-1.1/ivy/hadoop-minicluster-pom-template.xml?rev=1408468r1=1408467r2=1408468view=diff
==
--- hadoop/common/branches/branch-1.1/ivy/hadoop-minicluster-pom-template.xml
(original)
+++ hadoop/common/branches/branch-1.1/ivy/hadoop-minicluster-pom-template.xml
Mon Nov 12 21:44:06 2012
@@ -32,13 +32,13 @@
dependency
groupIdorg.apache.hadoop/groupId
artifactIdhadoop-core/artifactId
- version${project.version}/version
+ version@version/version
scopecompile/scope
/dependency
dependency
groupIdorg.apache.hadoop/groupId
artifactIdhadoop-test/artifactId
- version${project.version}/version
+ version@version/version
scopecompile/scope
/dependency
dependency
svn commit: r1408469 - in /hadoop/common/branches/branch-1: ./ CHANGES.txt ivy/hadoop-client-pom-template.xml ivy/hadoop-minicluster-pom-template.xml src/mapred/
Author: gkesavan
Date: Mon Nov 12 22:00:15 2012
New Revision: 1408469
URL: http://svn.apache.org/viewvc?rev=1408469view=rev
Log:
HADOOP-9017. svn merge -r 1408467:1408468 from branch-1.1
Modified:
hadoop/common/branches/branch-1/ (props changed)
hadoop/common/branches/branch-1/CHANGES.txt (contents, props changed)
hadoop/common/branches/branch-1/ivy/hadoop-client-pom-template.xml
hadoop/common/branches/branch-1/ivy/hadoop-minicluster-pom-template.xml
hadoop/common/branches/branch-1/src/mapred/ (props changed)
Propchange: hadoop/common/branches/branch-1/
--
Merged /hadoop/common/branches/branch-1.1:r1408468
Modified: hadoop/common/branches/branch-1/CHANGES.txt
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/CHANGES.txt?rev=1408469r1=1408468r2=1408469view=diff
==
--- hadoop/common/branches/branch-1/CHANGES.txt (original)
+++ hadoop/common/branches/branch-1/CHANGES.txt Mon Nov 12 22:00:15 2012
@@ -348,6 +348,10 @@ Release 1.1.1 - Unreleased
MAPREDUCE-4749. Fixed a bug in TaskTracker because of which kill-actions
get
delayed progressively. (Arpit Gupta via vinodkv)
+HADOOP-9017. Fix hadoop-client-pom-template.xml and
+hadoop-client-pom-template.xml for version token ( gkesavan )
+
+
Release 1.1.0 - 2012.09.28
INCOMPATIBLE CHANGES
Propchange: hadoop/common/branches/branch-1/CHANGES.txt
--
Merged /hadoop/common/branches/branch-1.1/CHANGES.txt:r1408468
Modified: hadoop/common/branches/branch-1/ivy/hadoop-client-pom-template.xml
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/ivy/hadoop-client-pom-template.xml?rev=1408469r1=1408468r2=1408469view=diff
==
--- hadoop/common/branches/branch-1/ivy/hadoop-client-pom-template.xml
(original)
+++ hadoop/common/branches/branch-1/ivy/hadoop-client-pom-template.xml Mon Nov
12 22:00:15 2012
@@ -32,7 +32,7 @@
dependency
groupIdorg.apache.hadoop/groupId
artifactIdhadoop-core/artifactId
- version${project.version}/version
+ version@version/version
scopecompile/scope
exclusions
exclusion
Modified:
hadoop/common/branches/branch-1/ivy/hadoop-minicluster-pom-template.xml
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/ivy/hadoop-minicluster-pom-template.xml?rev=1408469r1=1408468r2=1408469view=diff
==
--- hadoop/common/branches/branch-1/ivy/hadoop-minicluster-pom-template.xml
(original)
+++ hadoop/common/branches/branch-1/ivy/hadoop-minicluster-pom-template.xml Mon
Nov 12 22:00:15 2012
@@ -32,13 +32,13 @@
dependency
groupIdorg.apache.hadoop/groupId
artifactIdhadoop-core/artifactId
- version${project.version}/version
+ version@version/version
scopecompile/scope
/dependency
dependency
groupIdorg.apache.hadoop/groupId
artifactIdhadoop-test/artifactId
- version${project.version}/version
+ version@version/version
scopecompile/scope
/dependency
dependency
Propchange: hadoop/common/branches/branch-1/src/mapred/
--
Merged /hadoop/common/branches/branch-1.1/src/mapred:r1408468
svn commit: r1408473 - in /hadoop/common/trunk/hadoop-common-project/hadoop-common: ./ src/main/java/org/apache/hadoop/ipc/ src/main/java/org/apache/hadoop/security/ src/test/java/org/apache/hadoop/ip
Author: bobby
Date: Mon Nov 12 22:13:00 2012
New Revision: 1408473
URL: http://svn.apache.org/viewvc?rev=1408473view=rev
Log:
HADOOP-9021. Enforce configured SASL method on the server (daryn via bobby)
Modified:
hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestSaslRPC.java
Modified: hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
URL:
http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt?rev=1408473r1=1408472r2=1408473view=diff
==
--- hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
(original)
+++ hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt Mon Nov
12 22:13:00 2012
@@ -361,6 +361,9 @@ Release 2.0.3-alpha - Unreleased
HADOOP-8860. Split MapReduce and YARN sections in documentation navigation.
(tomwhite via tucu)
+HADOOP-9021. Enforce configured SASL method on the server (daryn via
+bobby)
+
OPTIMIZATIONS
HADOOP-8866. SampleQuantiles#query is O(N^2) instead of O(N). (Andrew Wang
Modified:
hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
URL:
http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java?rev=1408473r1=1408472r2=1408473view=diff
==
---
hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
(original)
+++
hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
Mon Nov 12 22:13:00 2012
@@ -45,6 +45,7 @@ import java.security.PrivilegedException
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
+import java.util.EnumSet;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Iterator;
@@ -87,7 +88,9 @@ import org.apache.hadoop.security.SaslRp
import org.apache.hadoop.security.SaslRpcServer.SaslDigestCallbackHandler;
import org.apache.hadoop.security.SaslRpcServer.SaslGssCallbackHandler;
import org.apache.hadoop.security.SaslRpcServer.SaslStatus;
+import org.apache.hadoop.security.SecurityUtil;
import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.UserGroupInformation.AuthenticationMethod;
import org.apache.hadoop.security.authentication.util.KerberosName;
import org.apache.hadoop.security.authorize.AuthorizationException;
import org.apache.hadoop.security.authorize.PolicyProvider;
@@ -113,7 +116,7 @@ import com.google.common.annotations.Vis
@InterfaceStability.Evolving
public abstract class Server {
private final boolean authorize;
- private boolean isSecurityEnabled;
+ private EnumSetAuthMethod enabledAuthMethods;
private ExceptionsHandler exceptionsHandler = new ExceptionsHandler();
public void addTerseExceptions(Class?... exceptionClass) {
@@ -1334,34 +1337,9 @@ public abstract class Server {
if (authMethod == null) {
throw new IOException(Unable to read authentication method);
}
- boolean useSaslServer = isSecurityEnabled;
- final boolean clientUsingSasl;
- switch (authMethod) {
-case SIMPLE: { // no sasl for simple
- clientUsingSasl = false;
- break;
-}
-case DIGEST: { // always allow tokens if there's a secret manager
- useSaslServer |= (secretManager != null);
- clientUsingSasl = true;
- break;
-}
-default: {
- clientUsingSasl = true;
- break;
-}
- }
- if (useSaslServer) {
-saslServer = createSaslServer(authMethod);
- } else if (clientUsingSasl) { // security is off
-doSaslReply(SaslStatus.SUCCESS, new IntWritable(
-SaslRpcServer.SWITCH_TO_SIMPLE_AUTH), null, null);
-authMethod = AuthMethod.SIMPLE;
-// client has already sent the initial Sasl message and we
-// should ignore it. Both client and server should fall back
-// to simple auth from now on.
-skipInitialSaslHandshake = true;
- }
+
+ // this may create a SASL server, or switch us into SIMPLE
+ authMethod = initializeAuthContext(authMethod);
connectionHeaderBuf = null;
connectionHeaderRead = true;
@@ -1409,10 +1387,24 @@ public abstract class
svn commit: r1408488 - /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
Author: bobby Date: Mon Nov 12 22:43:07 2012 New Revision: 1408488 URL: http://svn.apache.org/viewvc?rev=1408488view=rev Log: HADOOP-9022. Hadoop distcp tool fails to copy file if -m 0 specified (Jonathan Eagles vai bobby) Modified: hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt Modified: hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt?rev=1408488r1=1408487r2=1408488view=diff == --- hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt (original) +++ hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt Mon Nov 12 22:43:07 2012 @@ -1143,6 +1143,9 @@ Release 0.23.5 - UNRELEASED HADOOP-8986. Server$Call object is never released after it is sent (bobby) +HADOOP-9022. Hadoop distcp tool fails to copy file if -m 0 specified +(Jonathan Eagles vai bobby) + Release 0.23.4 - UNRELEASED INCOMPATIBLE CHANGES
svn commit: r1408491 - in /hadoop/common/branches/branch-2/hadoop-tools/hadoop-distcp/src: main/java/org/apache/hadoop/tools/DistCpOptions.java test/java/org/apache/hadoop/tools/TestOptionsParser.java
Author: bobby
Date: Mon Nov 12 22:47:05 2012
New Revision: 1408491
URL: http://svn.apache.org/viewvc?rev=1408491view=rev
Log:
svn merge -c 1408488 FIXES: HADOOP-9022. Hadoop distcp tool fails to copy file
if -m 0 specified (Jonathan Eagles vai bobby)
Modified:
hadoop/common/branches/branch-2/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
hadoop/common/branches/branch-2/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
Modified:
hadoop/common/branches/branch-2/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java?rev=1408491r1=1408490r2=1408491view=diff
==
---
hadoop/common/branches/branch-2/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
(original)
+++
hadoop/common/branches/branch-2/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
Mon Nov 12 22:47:05 2012
@@ -274,7 +274,7 @@ public class DistCpOptions {
* @param maxMaps - Number of maps
*/
public void setMaxMaps(int maxMaps) {
-this.maxMaps = maxMaps;
+this.maxMaps = Math.max(maxMaps, 1);
}
/** Get the map bandwidth in MB
Modified:
hadoop/common/branches/branch-2/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java?rev=1408491r1=1408490r2=1408491view=diff
==
---
hadoop/common/branches/branch-2/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
(original)
+++
hadoop/common/branches/branch-2/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
Mon Nov 12 22:47:05 2012
@@ -275,6 +275,13 @@ public class TestOptionsParser {
hdfs://localhost:8020/target/});
Assert.assertEquals(options.getMaxMaps(), 1);
+options = OptionsParser.parse(new String[] {
+-m,
+0,
+hdfs://localhost:8020/source/first,
+hdfs://localhost:8020/target/});
+Assert.assertEquals(options.getMaxMaps(), 1);
+
try {
OptionsParser.parse(new String[] {
-m,
svn commit: r1408491 - /hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/CHANGES.txt
Author: bobby Date: Mon Nov 12 22:47:05 2012 New Revision: 1408491 URL: http://svn.apache.org/viewvc?rev=1408491view=rev Log: svn merge -c 1408488 FIXES: HADOOP-9022. Hadoop distcp tool fails to copy file if -m 0 specified (Jonathan Eagles vai bobby) Modified: hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/CHANGES.txt Modified: hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/CHANGES.txt?rev=1408491r1=1408490r2=1408491view=diff == --- hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/CHANGES.txt (original) +++ hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/CHANGES.txt Mon Nov 12 22:47:05 2012 @@ -860,6 +860,9 @@ Release 0.23.5 - UNRELEASED HADOOP-8986. Server$Call object is never released after it is sent (bobby) +HADOOP-9022. Hadoop distcp tool fails to copy file if -m 0 specified +(Jonathan Eagles vai bobby) + Release 0.23.4 - UNRELEASED INCOMPATIBLE CHANGES
svn commit: r1408493 - in /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src: main/java/org/apache/hadoop/tools/DistCpOptions.java test/java/org/apache/hadoop/tools/TestOptionsParser.j
Author: bobby
Date: Mon Nov 12 22:52:01 2012
New Revision: 1408493
URL: http://svn.apache.org/viewvc?rev=1408493view=rev
Log:
svn merge -c 1408488 FIXES: HADOOP-9022. Hadoop distcp tool fails to copy file
if -m 0 specified (Jonathan Eagles vai bobby)
Modified:
hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
Modified:
hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java?rev=1408493r1=1408492r2=1408493view=diff
==
---
hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
(original)
+++
hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
Mon Nov 12 22:52:01 2012
@@ -274,7 +274,7 @@ public class DistCpOptions {
* @param maxMaps - Number of maps
*/
public void setMaxMaps(int maxMaps) {
-this.maxMaps = maxMaps;
+this.maxMaps = Math.max(maxMaps, 1);
}
/** Get the map bandwidth in MB
Modified:
hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java?rev=1408493r1=1408492r2=1408493view=diff
==
---
hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
(original)
+++
hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
Mon Nov 12 22:52:01 2012
@@ -275,6 +275,13 @@ public class TestOptionsParser {
hdfs://localhost:8020/target/});
Assert.assertEquals(options.getMaxMaps(), 1);
+options = OptionsParser.parse(new String[] {
+-m,
+0,
+hdfs://localhost:8020/source/first,
+hdfs://localhost:8020/target/});
+Assert.assertEquals(options.getMaxMaps(), 1);
+
try {
OptionsParser.parse(new String[] {
-m,
svn commit: r1408532 - /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/FsPermission.java
Author: eli
Date: Tue Nov 13 01:28:33 2012
New Revision: 1408532
URL: http://svn.apache.org/viewvc?rev=1408532view=rev
Log:
HDFS-1322. Document umask in DistributedFileSystem#mkdirs javadocs. Contributed
by Colin Patrick McCabe
Modified:
hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/FsPermission.java
Modified:
hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/FsPermission.java
URL:
http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/FsPermission.java?rev=1408532r1=1408531r2=1408532view=diff
==
---
hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/FsPermission.java
(original)
+++
hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/FsPermission.java
Tue Nov 13 01:28:33 2012
@@ -184,7 +184,18 @@ public class FsPermission implements Wri
return str;
}
- /** Apply a umask to this permission and return a new one */
+ /**
+ * Apply a umask to this permission and return a new one.
+ *
+ * The umask is used by create, mkdir, and other Hadoop filesystem
operations.
+ * The mode argument for these operations is modified by removing the bits
+ * which are set in the umask. Thus, the umask limits the permissions which
+ * newly created files and directories get.
+ *
+ * @param umask The umask to use
+ *
+ * @return The effective permission
+ */
public FsPermission applyUMask(FsPermission umask) {
return new FsPermission(useraction.and(umask.useraction.not()),
groupaction.and(umask.groupaction.not()),
svn commit: r1408533 - /hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/FsPermission.java
Author: eli
Date: Tue Nov 13 01:29:51 2012
New Revision: 1408533
URL: http://svn.apache.org/viewvc?rev=1408533view=rev
Log:
HDFS-1322. Document umask in DistributedFileSystem#mkdirs javadocs. Contributed
by Colin Patrick McCabe
Modified:
hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/FsPermission.java
Modified:
hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/FsPermission.java
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/FsPermission.java?rev=1408533r1=1408532r2=1408533view=diff
==
---
hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/FsPermission.java
(original)
+++
hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/FsPermission.java
Tue Nov 13 01:29:51 2012
@@ -184,7 +184,18 @@ public class FsPermission implements Wri
return str;
}
- /** Apply a umask to this permission and return a new one */
+ /**
+ * Apply a umask to this permission and return a new one.
+ *
+ * The umask is used by create, mkdir, and other Hadoop filesystem
operations.
+ * The mode argument for these operations is modified by removing the bits
+ * which are set in the umask. Thus, the umask limits the permissions which
+ * newly created files and directories get.
+ *
+ * @param umask The umask to use
+ *
+ * @return The effective permission
+ */
public FsPermission applyUMask(FsPermission umask) {
return new FsPermission(useraction.and(umask.useraction.not()),
groupaction.and(umask.groupaction.not()),
svn commit: r1408604 - /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java
Author: jlowe Date: Tue Nov 13 05:53:26 2012 New Revision: 1408604 URL: http://svn.apache.org/viewvc?rev=1408604view=rev Log: HADOOP-9025. org.apache.hadoop.tools.TestCopyListing failing. Contributed by Jonathan Eagles Modified: hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java Modified: hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java URL: http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java?rev=1408604r1=1408603r2=1408604view=diff == --- hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java (original) +++ hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java Tue Nov 13 05:53:26 2012 @@ -131,8 +131,8 @@ public class TestCopyListing extends Sim fs = FileSystem.get(getConf()); ListPath srcPaths = new ArrayListPath(); srcPaths.add(new Path(/tmp/in/*/*)); - TestDistCpUtils.createFile(fs, /tmp/in/1.txt); - TestDistCpUtils.createFile(fs, /tmp/in/src/1.txt); + TestDistCpUtils.createFile(fs, /tmp/in/src1/1.txt); + TestDistCpUtils.createFile(fs, /tmp/in/src2/1.txt); Path target = new Path(/tmp/out); Path listingFile = new Path(/tmp/list); DistCpOptions options = new DistCpOptions(srcPaths, target);
svn commit: r1408604 - /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
Author: jlowe Date: Tue Nov 13 05:53:26 2012 New Revision: 1408604 URL: http://svn.apache.org/viewvc?rev=1408604view=rev Log: HADOOP-9025. org.apache.hadoop.tools.TestCopyListing failing. Contributed by Jonathan Eagles Modified: hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt Modified: hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt?rev=1408604r1=1408603r2=1408604view=diff == --- hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt (original) +++ hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt Tue Nov 13 05:53:26 2012 @@ -1146,6 +1146,9 @@ Release 0.23.5 - UNRELEASED HADOOP-9022. Hadoop distcp tool fails to copy file if -m 0 specified (Jonathan Eagles vai bobby) +HADOOP-9025. org.apache.hadoop.tools.TestCopyListing failing (Jonathan +Eagles via jlowe) + Release 0.23.4 - UNRELEASED INCOMPATIBLE CHANGES
svn commit: r1408605 - /hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/CHANGES.txt
Author: jlowe Date: Tue Nov 13 05:55:51 2012 New Revision: 1408605 URL: http://svn.apache.org/viewvc?rev=1408605view=rev Log: svn merge -c 1408604 FIXES: HADOOP-9025. org.apache.hadoop.tools.TestCopyListing failing. Contributed by Jonathan Eagles Modified: hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/CHANGES.txt Modified: hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/CHANGES.txt?rev=1408605r1=1408604r2=1408605view=diff == --- hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/CHANGES.txt (original) +++ hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/CHANGES.txt Tue Nov 13 05:55:51 2012 @@ -863,6 +863,9 @@ Release 0.23.5 - UNRELEASED HADOOP-9022. Hadoop distcp tool fails to copy file if -m 0 specified (Jonathan Eagles vai bobby) +HADOOP-9025. org.apache.hadoop.tools.TestCopyListing failing (Jonathan +Eagles via jlowe) + Release 0.23.4 - UNRELEASED INCOMPATIBLE CHANGES
svn commit: r1408605 - /hadoop/common/branches/branch-2/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java
Author: jlowe Date: Tue Nov 13 05:55:51 2012 New Revision: 1408605 URL: http://svn.apache.org/viewvc?rev=1408605view=rev Log: svn merge -c 1408604 FIXES: HADOOP-9025. org.apache.hadoop.tools.TestCopyListing failing. Contributed by Jonathan Eagles Modified: hadoop/common/branches/branch-2/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java Modified: hadoop/common/branches/branch-2/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java?rev=1408605r1=1408604r2=1408605view=diff == --- hadoop/common/branches/branch-2/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java (original) +++ hadoop/common/branches/branch-2/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java Tue Nov 13 05:55:51 2012 @@ -131,8 +131,8 @@ public class TestCopyListing extends Sim fs = FileSystem.get(getConf()); ListPath srcPaths = new ArrayListPath(); srcPaths.add(new Path(/tmp/in/*/*)); - TestDistCpUtils.createFile(fs, /tmp/in/1.txt); - TestDistCpUtils.createFile(fs, /tmp/in/src/1.txt); + TestDistCpUtils.createFile(fs, /tmp/in/src1/1.txt); + TestDistCpUtils.createFile(fs, /tmp/in/src2/1.txt); Path target = new Path(/tmp/out); Path listingFile = new Path(/tmp/list); DistCpOptions options = new DistCpOptions(srcPaths, target);
svn commit: r1408606 - /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
Author: jlowe Date: Tue Nov 13 05:56:39 2012 New Revision: 1408606 URL: http://svn.apache.org/viewvc?rev=1408606view=rev Log: svn merge -c 1408604 FIXES: HADOOP-9025. org.apache.hadoop.tools.TestCopyListing failing. Contributed by Jonathan Eagles Modified: hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt Modified: hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt?rev=1408606r1=1408605r2=1408606view=diff == --- hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt (original) +++ hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt Tue Nov 13 05:56:39 2012 @@ -58,6 +58,9 @@ Release 0.23.5 - UNRELEASED HADOOP-9022. Hadoop distcp tool fails to copy file if -m 0 specified (Jonathan Eagles vai bobby) +HADOOP-9025. org.apache.hadoop.tools.TestCopyListing failing (Jonathan +Eagles via jlowe) + Release 0.23.4 - UNRELEASED INCOMPATIBLE CHANGES
svn commit: r1408606 - /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java
Author: jlowe Date: Tue Nov 13 05:56:39 2012 New Revision: 1408606 URL: http://svn.apache.org/viewvc?rev=1408606view=rev Log: svn merge -c 1408604 FIXES: HADOOP-9025. org.apache.hadoop.tools.TestCopyListing failing. Contributed by Jonathan Eagles Modified: hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java Modified: hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java?rev=1408606r1=1408605r2=1408606view=diff == --- hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java (original) +++ hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java Tue Nov 13 05:56:39 2012 @@ -131,8 +131,8 @@ public class TestCopyListing extends Sim fs = FileSystem.get(getConf()); ListPath srcPaths = new ArrayListPath(); srcPaths.add(new Path(/tmp/in/*/*)); - TestDistCpUtils.createFile(fs, /tmp/in/1.txt); - TestDistCpUtils.createFile(fs, /tmp/in/src/1.txt); + TestDistCpUtils.createFile(fs, /tmp/in/src1/1.txt); + TestDistCpUtils.createFile(fs, /tmp/in/src2/1.txt); Path target = new Path(/tmp/out); Path listingFile = new Path(/tmp/list); DistCpOptions options = new DistCpOptions(srcPaths, target);
