hadoop git commit: HADOOP-15296. Fix a wrong link for RBF in the top page. Contributed by Takanobu Asanuma.
Repository: hadoop Updated Branches: refs/heads/trunk 583f45943 -> 4cc9a6d9b HADOOP-15296. Fix a wrong link for RBF in the top page. Contributed by Takanobu Asanuma. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/4cc9a6d9 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/4cc9a6d9 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/4cc9a6d9 Branch: refs/heads/trunk Commit: 4cc9a6d9bb34329d6de30706d5432c7cb675bb88 Parents: 583f459 Author: Yiqun Lin Authored: Thu Mar 8 16:02:34 2018 +0800 Committer: Yiqun Lin Committed: Thu Mar 8 16:02:34 2018 +0800 -- hadoop-project/src/site/markdown/index.md.vm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/4cc9a6d9/hadoop-project/src/site/markdown/index.md.vm -- diff --git a/hadoop-project/src/site/markdown/index.md.vm b/hadoop-project/src/site/markdown/index.md.vm index 9b2d9de..8b9cfda 100644 --- a/hadoop-project/src/site/markdown/index.md.vm +++ b/hadoop-project/src/site/markdown/index.md.vm @@ -223,7 +223,7 @@ functionality, except the mount table is managed on the server-side by the routing layer rather than on the client. This simplifies access to a federated cluster for existing HDFS clients. -See [HDFS-10467](https://issues.apache.org/jira/browse/HADOOP-10467) and the +See [HDFS-10467](https://issues.apache.org/jira/browse/HDFS-10467) and the HDFS Router-based Federation [documentation](./hadoop-project-dist/hadoop-hdfs/HDFSRouterFederation.html) for more details. - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HADOOP-15296. Fix a wrong link for RBF in the top page. Contributed by Takanobu Asanuma.
Repository: hadoop Updated Branches: refs/heads/branch-3.1 885d46a24 -> 56f14cb5f HADOOP-15296. Fix a wrong link for RBF in the top page. Contributed by Takanobu Asanuma. (cherry picked from commit 4cc9a6d9bb34329d6de30706d5432c7cb675bb88) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/56f14cb5 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/56f14cb5 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/56f14cb5 Branch: refs/heads/branch-3.1 Commit: 56f14cb5fb6efe1f518d9202506f7b68f6912b7f Parents: 885d46a Author: Yiqun Lin Authored: Thu Mar 8 16:02:34 2018 +0800 Committer: Yiqun Lin Committed: Thu Mar 8 16:04:20 2018 +0800 -- hadoop-project/src/site/markdown/index.md.vm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/56f14cb5/hadoop-project/src/site/markdown/index.md.vm -- diff --git a/hadoop-project/src/site/markdown/index.md.vm b/hadoop-project/src/site/markdown/index.md.vm index 9b2d9de..8b9cfda 100644 --- a/hadoop-project/src/site/markdown/index.md.vm +++ b/hadoop-project/src/site/markdown/index.md.vm @@ -223,7 +223,7 @@ functionality, except the mount table is managed on the server-side by the routing layer rather than on the client. This simplifies access to a federated cluster for existing HDFS clients. -See [HDFS-10467](https://issues.apache.org/jira/browse/HADOOP-10467) and the +See [HDFS-10467](https://issues.apache.org/jira/browse/HDFS-10467) and the HDFS Router-based Federation [documentation](./hadoop-project-dist/hadoop-hdfs/HDFSRouterFederation.html) for more details. - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HADOOP-15296. Fix a wrong link for RBF in the top page. Contributed by Takanobu Asanuma.
Repository: hadoop Updated Branches: refs/heads/branch-3.0 0b1e9665d -> 7a9aea17a HADOOP-15296. Fix a wrong link for RBF in the top page. Contributed by Takanobu Asanuma. (cherry picked from commit 4cc9a6d9bb34329d6de30706d5432c7cb675bb88) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/7a9aea17 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/7a9aea17 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/7a9aea17 Branch: refs/heads/branch-3.0 Commit: 7a9aea17a2ecd5f9f42d6eef792c7d9a8f310e51 Parents: 0b1e966 Author: Yiqun Lin Authored: Thu Mar 8 16:02:34 2018 +0800 Committer: Yiqun Lin Committed: Thu Mar 8 16:05:53 2018 +0800 -- hadoop-project/src/site/markdown/index.md.vm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/7a9aea17/hadoop-project/src/site/markdown/index.md.vm -- diff --git a/hadoop-project/src/site/markdown/index.md.vm b/hadoop-project/src/site/markdown/index.md.vm index 9b2d9de..8b9cfda 100644 --- a/hadoop-project/src/site/markdown/index.md.vm +++ b/hadoop-project/src/site/markdown/index.md.vm @@ -223,7 +223,7 @@ functionality, except the mount table is managed on the server-side by the routing layer rather than on the client. This simplifies access to a federated cluster for existing HDFS clients. -See [HDFS-10467](https://issues.apache.org/jira/browse/HADOOP-10467) and the +See [HDFS-10467](https://issues.apache.org/jira/browse/HDFS-10467) and the HDFS Router-based Federation [documentation](./hadoop-project-dist/hadoop-hdfs/HDFSRouterFederation.html) for more details. - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: YARN-8011. TestOpportunisticContainerAllocatorAMService#testContainerPromoteAndDemoteBeforeContainerStart fails intermittently. Contributed by Tao Yang.
Repository: hadoop
Updated Branches:
refs/heads/trunk 4cc9a6d9b -> b451889e8
YARN-8011.
TestOpportunisticContainerAllocatorAMService#testContainerPromoteAndDemoteBeforeContainerStart
fails intermittently. Contributed by Tao Yang.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/b451889e
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/b451889e
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/b451889e
Branch: refs/heads/trunk
Commit: b451889e8e83f7977f2b76789c61e823e2d40487
Parents: 4cc9a6d
Author: Weiwei Yang
Authored: Thu Mar 8 18:13:36 2018 +0800
Committer: Weiwei Yang
Committed: Thu Mar 8 18:13:36 2018 +0800
--
...pportunisticContainerAllocatorAMService.java | 29 ++--
1 file changed, 15 insertions(+), 14 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b451889e/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestOpportunisticContainerAllocatorAMService.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestOpportunisticContainerAllocatorAMService.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestOpportunisticContainerAllocatorAMService.java
index 1af930f..efa76bc 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestOpportunisticContainerAllocatorAMService.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestOpportunisticContainerAllocatorAMService.java
@@ -243,13 +243,13 @@ public class TestOpportunisticContainerAllocatorAMService
{
null, ExecutionType.GUARANTEED)));
// Node on same host should not result in allocation
sameHostDiffNode.nodeHeartbeat(true);
-Thread.sleep(200);
+rm.drainEvents();
allocateResponse = am1.allocate(new ArrayList<>(), new ArrayList<>());
Assert.assertEquals(0, allocateResponse.getUpdatedContainers().size());
// Wait for scheduler to process all events
dispatcher.waitForEventThreadToWait();
-Thread.sleep(1000);
+rm.drainEvents();
// Verify Metrics After OPP allocation (Nothing should change again)
verifyMetrics(metrics, 15360, 15, 1024, 1, 1);
@@ -286,7 +286,7 @@ public class TestOpportunisticContainerAllocatorAMService {
// Ensure after correct node heartbeats, we should get the allocation
allocNode.nodeHeartbeat(true);
-Thread.sleep(200);
+rm.drainEvents();
allocateResponse = am1.allocate(new ArrayList<>(), new ArrayList<>());
Assert.assertEquals(1, allocateResponse.getUpdatedContainers().size());
Container uc =
@@ -303,7 +303,7 @@ public class TestOpportunisticContainerAllocatorAMService {
nm2.nodeHeartbeat(true);
nm3.nodeHeartbeat(true);
nm4.nodeHeartbeat(true);
-Thread.sleep(200);
+rm.drainEvents();
// Verify that the container is still in ACQUIRED state wrt the RM.
RMContainer rmContainer = ((CapacityScheduler) scheduler)
@@ -325,6 +325,7 @@ public class TestOpportunisticContainerAllocatorAMService {
// Wait for scheduler to finish processing events
dispatcher.waitForEventThreadToWait();
+rm.drainEvents();
// Verify Metrics After OPP allocation :
// Everything should have reverted to what it was
verifyMetrics(metrics, 15360, 15, 1024, 1, 1);
@@ -396,7 +397,7 @@ public class TestOpportunisticContainerAllocatorAMService {
ContainerStatus.newInstance(container.getId(),
ExecutionType.OPPORTUNISTIC, ContainerState.RUNNING, "", 0)),
true);
-Thread.sleep(200);
+rm.drainEvents();
// Verify that container is actually running wrt the RM..
RMContainer rmContainer = ((CapacityScheduler) scheduler)
@@ -434,7 +435,7 @@ public class TestOpportunisticContainerAllocatorAMService {
ContainerStatus.newInstance(container.getId(),
ExecutionType.OPPORTUNISTIC, ContainerState.RUNNING, "", 0)),
true);
-Thread.sleep(200);
+rm.drainEvents();
allocateResponse = am1.allocate(new ArrayList<>(), new ArrayList<>());
Assert.assertEquals(1, allocateResponse.getUpdatedContainers().size());
@@ -521,7 +522,7 @@ public class TestOpportunisticContainerAllocatorAMService {
ContainerStatus.newInstance(container.getId(),
ExecutionType.OPPORTUNISTI
[2/2] hadoop git commit: HADOOP-15292. Distcp's use of pread is slowing it down. Contributed by Virajith Jalaparti.
HADOOP-15292. Distcp's use of pread is slowing it down.
Contributed by Virajith Jalaparti.
(cherry picked from commit 3bd6b1fd85c44354c777ef4fda6415231505b2a4)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/f0b486f6
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/f0b486f6
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/f0b486f6
Branch: refs/heads/branch-3.1
Commit: f0b486f6ae2cc76c7cf80a0681a16386c00d3390
Parents: 56f14cb
Author: Steve Loughran
Authored: Thu Mar 8 11:18:33 2018 +
Committer: Steve Loughran
Committed: Thu Mar 8 11:18:33 2018 +
--
.../tools/mapred/RetriableFileCopyCommand.java | 24 ++
.../hadoop/tools/util/ThrottledInputStream.java | 48 +++-
.../hadoop/tools/mapred/TestCopyMapper.java | 24 +-
3 files changed, 66 insertions(+), 30 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/f0b486f6/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
--
diff --git
a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
index 21f621a..0311061 100644
---
a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
+++
b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
@@ -260,7 +260,8 @@ public class RetriableFileCopyCommand extends
RetriableCommand {
boolean finished = false;
try {
inStream = getInputStream(source, context.getConfiguration());
- int bytesRead = readBytes(inStream, buf, sourceOffset);
+ seekIfRequired(inStream, sourceOffset);
+ int bytesRead = readBytes(inStream, buf);
while (bytesRead >= 0) {
if (chunkLength > 0 &&
(totalBytesRead + bytesRead) >= chunkLength) {
@@ -276,7 +277,7 @@ public class RetriableFileCopyCommand extends
RetriableCommand {
if (finished) {
break;
}
-bytesRead = readBytes(inStream, buf, sourceOffset);
+bytesRead = readBytes(inStream, buf);
}
outStream.close();
outStream = null;
@@ -299,13 +300,20 @@ public class RetriableFileCopyCommand extends
RetriableCommand {
context.setStatus(message.toString());
}
- private static int readBytes(ThrottledInputStream inStream, byte buf[],
- long position) throws IOException {
+ private static int readBytes(ThrottledInputStream inStream, byte buf[])
+ throws IOException {
+try {
+ return inStream.read(buf);
+} catch (IOException e) {
+ throw new CopyReadException(e);
+}
+ }
+
+ private static void seekIfRequired(ThrottledInputStream inStream,
+ long sourceOffset) throws IOException {
try {
- if (position == 0) {
-return inStream.read(buf);
- } else {
-return inStream.read(position, buf, 0, buf.length);
+ if (sourceOffset != inStream.getPos()) {
+inStream.seek(sourceOffset);
}
} catch (IOException e) {
throw new CopyReadException(e);
http://git-wip-us.apache.org/repos/asf/hadoop/blob/f0b486f6/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/ThrottledInputStream.java
--
diff --git
a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/ThrottledInputStream.java
b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/ThrottledInputStream.java
index 2d2f10c..4d3676a 100644
---
a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/ThrottledInputStream.java
+++
b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/ThrottledInputStream.java
@@ -18,7 +18,7 @@
package org.apache.hadoop.tools.util;
-import org.apache.hadoop.fs.PositionedReadable;
+import org.apache.hadoop.fs.Seekable;
import java.io.IOException;
import java.io.InputStream;
@@ -33,7 +33,7 @@ import java.io.InputStream;
* (Thus, while the read-rate might exceed the maximum for a given short
interval,
* the average tends towards the specified maximum, overall.)
*/
-public class ThrottledInputStream extends InputStream {
+public class ThrottledInputStream extends InputStream implements Seekable {
private final InputStream rawStream;
private final float maxBytesPerSec;
@@ -95,25 +95,6 @@ public class ThrottledInputStream extends InputStream {
return readLen;
}
- /**
- * Read bytes starting from the specified position. This requires rawStream
is
- * an instance of {@link PositionedR
[1/2] hadoop git commit: HADOOP-15292. Distcp's use of pread is slowing it down. Contributed by Virajith Jalaparti.
Repository: hadoop
Updated Branches:
refs/heads/branch-3.1 56f14cb5f -> f0b486f6a
refs/heads/trunk b451889e8 -> 3bd6b1fd8
HADOOP-15292. Distcp's use of pread is slowing it down.
Contributed by Virajith Jalaparti.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/3bd6b1fd
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/3bd6b1fd
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/3bd6b1fd
Branch: refs/heads/trunk
Commit: 3bd6b1fd85c44354c777ef4fda6415231505b2a4
Parents: b451889
Author: Steve Loughran
Authored: Thu Mar 8 11:15:46 2018 +
Committer: Steve Loughran
Committed: Thu Mar 8 11:15:46 2018 +
--
.../tools/mapred/RetriableFileCopyCommand.java | 24 ++
.../hadoop/tools/util/ThrottledInputStream.java | 48 +++-
.../hadoop/tools/mapred/TestCopyMapper.java | 24 +-
3 files changed, 66 insertions(+), 30 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/3bd6b1fd/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
--
diff --git
a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
index 21f621a..0311061 100644
---
a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
+++
b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
@@ -260,7 +260,8 @@ public class RetriableFileCopyCommand extends
RetriableCommand {
boolean finished = false;
try {
inStream = getInputStream(source, context.getConfiguration());
- int bytesRead = readBytes(inStream, buf, sourceOffset);
+ seekIfRequired(inStream, sourceOffset);
+ int bytesRead = readBytes(inStream, buf);
while (bytesRead >= 0) {
if (chunkLength > 0 &&
(totalBytesRead + bytesRead) >= chunkLength) {
@@ -276,7 +277,7 @@ public class RetriableFileCopyCommand extends
RetriableCommand {
if (finished) {
break;
}
-bytesRead = readBytes(inStream, buf, sourceOffset);
+bytesRead = readBytes(inStream, buf);
}
outStream.close();
outStream = null;
@@ -299,13 +300,20 @@ public class RetriableFileCopyCommand extends
RetriableCommand {
context.setStatus(message.toString());
}
- private static int readBytes(ThrottledInputStream inStream, byte buf[],
- long position) throws IOException {
+ private static int readBytes(ThrottledInputStream inStream, byte buf[])
+ throws IOException {
+try {
+ return inStream.read(buf);
+} catch (IOException e) {
+ throw new CopyReadException(e);
+}
+ }
+
+ private static void seekIfRequired(ThrottledInputStream inStream,
+ long sourceOffset) throws IOException {
try {
- if (position == 0) {
-return inStream.read(buf);
- } else {
-return inStream.read(position, buf, 0, buf.length);
+ if (sourceOffset != inStream.getPos()) {
+inStream.seek(sourceOffset);
}
} catch (IOException e) {
throw new CopyReadException(e);
http://git-wip-us.apache.org/repos/asf/hadoop/blob/3bd6b1fd/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/ThrottledInputStream.java
--
diff --git
a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/ThrottledInputStream.java
b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/ThrottledInputStream.java
index 2d2f10c..4d3676a 100644
---
a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/ThrottledInputStream.java
+++
b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/ThrottledInputStream.java
@@ -18,7 +18,7 @@
package org.apache.hadoop.tools.util;
-import org.apache.hadoop.fs.PositionedReadable;
+import org.apache.hadoop.fs.Seekable;
import java.io.IOException;
import java.io.InputStream;
@@ -33,7 +33,7 @@ import java.io.InputStream;
* (Thus, while the read-rate might exceed the maximum for a given short
interval,
* the average tends towards the specified maximum, overall.)
*/
-public class ThrottledInputStream extends InputStream {
+public class ThrottledInputStream extends InputStream implements Seekable {
private final InputStream rawStream;
private final float maxBytesPerSec;
@@ -95,25 +95,6 @@ public class ThrottledInputStream extends InputStream {
return readLen;
}
- /**
- * Read bytes starting from the specified position. This requires
[1/2] hadoop git commit: HADOOP-15273.distcp can't handle remote stores with different checksum algorithms. Contributed by Steve Loughran.
Repository: hadoop
Updated Branches:
refs/heads/branch-3.1 f0b486f6a -> ba0184376
refs/heads/trunk 3bd6b1fd8 -> 7ef4d942d
HADOOP-15273.distcp can't handle remote stores with different checksum
algorithms.
Contributed by Steve Loughran.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/7ef4d942
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/7ef4d942
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/7ef4d942
Branch: refs/heads/trunk
Commit: 7ef4d942dd96232b0743a40ed25f77065254f94d
Parents: 3bd6b1f
Author: Steve Loughran
Authored: Thu Mar 8 11:24:06 2018 +
Committer: Steve Loughran
Committed: Thu Mar 8 11:24:06 2018 +
--
.../org/apache/hadoop/tools/DistCpOptions.java | 5
.../tools/mapred/RetriableFileCopyCommand.java | 29 +++-
.../hadoop/tools/mapred/TestCopyMapper.java | 14 +-
3 files changed, 29 insertions(+), 19 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/7ef4d942/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
--
diff --git
a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
index ece1a94..f33f7fd 100644
---
a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
+++
b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
@@ -534,11 +534,6 @@ public final class DistCpOptions {
+ "mutually exclusive");
}
- if (!syncFolder && skipCRC) {
-throw new IllegalArgumentException(
-"Skip CRC is valid only with update options");
- }
-
if (!syncFolder && append) {
throw new IllegalArgumentException(
"Append is valid only with update options");
http://git-wip-us.apache.org/repos/asf/hadoop/blob/7ef4d942/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
--
diff --git
a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
index 0311061..55f90d0 100644
---
a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
+++
b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
@@ -210,15 +210,30 @@ public class RetriableFileCopyCommand extends
RetriableCommand {
throws IOException {
if (!DistCpUtils.checksumsAreEqual(sourceFS, source, sourceChecksum,
targetFS, target)) {
- StringBuilder errorMessage = new StringBuilder("Check-sum mismatch
between ")
- .append(source).append(" and ").append(target).append(".");
- if (sourceFS.getFileStatus(source).getBlockSize() !=
+ StringBuilder errorMessage =
+ new StringBuilder("Checksum mismatch between ")
+ .append(source).append(" and ").append(target).append(".");
+ boolean addSkipHint = false;
+ String srcScheme = sourceFS.getScheme();
+ String targetScheme = targetFS.getScheme();
+ if (!srcScheme.equals(targetScheme)
+ && !(srcScheme.contains("hdfs") && targetScheme.contains("hdfs"))) {
+// the filesystems are different and they aren't both hdfs connectors
+errorMessage.append("Source and destination filesystems are of"
++ " different types\n")
+.append("Their checksum algorithms may be incompatible");
+addSkipHint = true;
+ } else if (sourceFS.getFileStatus(source).getBlockSize() !=
targetFS.getFileStatus(target).getBlockSize()) {
-errorMessage.append(" Source and target differ in block-size.")
-.append(" Use -pb to preserve block-sizes during copy.")
-.append(" Alternatively, skip checksum-checks altogether, using
-skipCrc.")
+errorMessage.append(" Source and target differ in block-size.\n")
+.append(" Use -pb to preserve block-sizes during copy.");
+addSkipHint = true;
+ }
+ if (addSkipHint) {
+errorMessage.append(" You can skip checksum-checks altogether "
++ " with -skipcrccheck.\n")
.append(" (NOTE: By skipping checksums, one runs the risk of " +
-"masking data-corruption during file-transfer.)");
+"masking data-corruption during file-transfer.)\n");
}
throw new IOException(errorMessage.toString());
}
http://git-wip-us.apache.org/rep
[2/2] hadoop git commit: HADOOP-15273.distcp can't handle remote stores with different checksum algorithms. Contributed by Steve Loughran.
HADOOP-15273.distcp can't handle remote stores with different checksum
algorithms.
Contributed by Steve Loughran.
(cherry picked from commit 7ef4d942dd96232b0743a40ed25f77065254f94d)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/ba018437
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/ba018437
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/ba018437
Branch: refs/heads/branch-3.1
Commit: ba0184376e8b9e46657f6c6557d694529bb3ceb0
Parents: f0b486f
Author: Steve Loughran
Authored: Thu Mar 8 11:24:06 2018 +
Committer: Steve Loughran
Committed: Thu Mar 8 11:25:05 2018 +
--
.../org/apache/hadoop/tools/DistCpOptions.java | 5
.../tools/mapred/RetriableFileCopyCommand.java | 29 +++-
.../hadoop/tools/mapred/TestCopyMapper.java | 14 +-
3 files changed, 29 insertions(+), 19 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/ba018437/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
--
diff --git
a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
index ece1a94..f33f7fd 100644
---
a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
+++
b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
@@ -534,11 +534,6 @@ public final class DistCpOptions {
+ "mutually exclusive");
}
- if (!syncFolder && skipCRC) {
-throw new IllegalArgumentException(
-"Skip CRC is valid only with update options");
- }
-
if (!syncFolder && append) {
throw new IllegalArgumentException(
"Append is valid only with update options");
http://git-wip-us.apache.org/repos/asf/hadoop/blob/ba018437/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
--
diff --git
a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
index 0311061..55f90d0 100644
---
a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
+++
b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
@@ -210,15 +210,30 @@ public class RetriableFileCopyCommand extends
RetriableCommand {
throws IOException {
if (!DistCpUtils.checksumsAreEqual(sourceFS, source, sourceChecksum,
targetFS, target)) {
- StringBuilder errorMessage = new StringBuilder("Check-sum mismatch
between ")
- .append(source).append(" and ").append(target).append(".");
- if (sourceFS.getFileStatus(source).getBlockSize() !=
+ StringBuilder errorMessage =
+ new StringBuilder("Checksum mismatch between ")
+ .append(source).append(" and ").append(target).append(".");
+ boolean addSkipHint = false;
+ String srcScheme = sourceFS.getScheme();
+ String targetScheme = targetFS.getScheme();
+ if (!srcScheme.equals(targetScheme)
+ && !(srcScheme.contains("hdfs") && targetScheme.contains("hdfs"))) {
+// the filesystems are different and they aren't both hdfs connectors
+errorMessage.append("Source and destination filesystems are of"
++ " different types\n")
+.append("Their checksum algorithms may be incompatible");
+addSkipHint = true;
+ } else if (sourceFS.getFileStatus(source).getBlockSize() !=
targetFS.getFileStatus(target).getBlockSize()) {
-errorMessage.append(" Source and target differ in block-size.")
-.append(" Use -pb to preserve block-sizes during copy.")
-.append(" Alternatively, skip checksum-checks altogether, using
-skipCrc.")
+errorMessage.append(" Source and target differ in block-size.\n")
+.append(" Use -pb to preserve block-sizes during copy.");
+addSkipHint = true;
+ }
+ if (addSkipHint) {
+errorMessage.append(" You can skip checksum-checks altogether "
++ " with -skipcrccheck.\n")
.append(" (NOTE: By skipping checksums, one runs the risk of " +
-"masking data-corruption during file-transfer.)");
+"masking data-corruption during file-transfer.)\n");
}
throw new IOException(errorMessage.toString());
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/ba018437/hadoop-tools/hadoop-distc
hadoop git commit: HDFS-13232. RBF: ConnectionManager's cleanup task will compare each pool's own active conns with its total conns. Contributed by Chao Sun.
Repository: hadoop
Updated Branches:
refs/heads/trunk 7ef4d942d -> 0c2b969e0
HDFS-13232. RBF: ConnectionManager's cleanup task will compare each pool's own
active conns with its total conns. Contributed by Chao Sun.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/0c2b969e
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/0c2b969e
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/0c2b969e
Branch: refs/heads/trunk
Commit: 0c2b969e0161a068bf9ae013c4b95508dfb90a8a
Parents: 7ef4d94
Author: Inigo Goiri
Authored: Thu Mar 8 09:32:05 2018 -0800
Committer: Inigo Goiri
Committed: Thu Mar 8 09:32:05 2018 -0800
--
.../federation/router/ConnectionManager.java| 59 +-
.../federation/router/ConnectionPoolId.java | 6 +
.../router/TestConnectionManager.java | 114 +++
3 files changed, 153 insertions(+), 26 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/0c2b969e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java
index 2e45280..594f489 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java
@@ -32,6 +32,7 @@ import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReadWriteLock;
import java.util.concurrent.locks.ReentrantReadWriteLock;
+import com.google.common.annotations.VisibleForTesting;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hdfs.DFSConfigKeys;
import org.apache.hadoop.security.UserGroupInformation;
@@ -303,6 +304,38 @@ public class ConnectionManager {
return JSON.toString(info);
}
+ @VisibleForTesting
+ Map getPools() {
+return this.pools;
+ }
+
+ /**
+ * Clean the unused connections for this pool.
+ *
+ * @param pool Connection pool to cleanup.
+ */
+ @VisibleForTesting
+ void cleanup(ConnectionPool pool) {
+if (pool.getNumConnections() > pool.getMinSize()) {
+ // Check if the pool hasn't been active in a while or not 50% are used
+ long timeSinceLastActive = Time.now() - pool.getLastActiveTime();
+ int total = pool.getNumConnections();
+ int active = pool.getNumActiveConnections();
+ if (timeSinceLastActive > connectionCleanupPeriodMs ||
+ active < MIN_ACTIVE_RATIO * total) {
+// Remove and close 1 connection
+List conns = pool.removeConnections(1);
+for (ConnectionContext conn : conns) {
+ conn.close();
+}
+LOG.debug("Removed connection {} used {} seconds ago. " +
+"Pool has {}/{} connections", pool.getConnectionPoolId(),
+TimeUnit.MILLISECONDS.toSeconds(timeSinceLastActive),
+pool.getNumConnections(), pool.getMaxSize());
+ }
+}
+ }
+
/**
* Removes stale connections not accessed recently from the pool. This is
* invoked periodically.
@@ -350,32 +383,6 @@ public class ConnectionManager {
}
}
}
-
-/**
- * Clean the unused connections for this pool.
- *
- * @param pool Connection pool to cleanup.
- */
-private void cleanup(ConnectionPool pool) {
- if (pool.getNumConnections() > pool.getMinSize()) {
-// Check if the pool hasn't been active in a while or not 50% are used
-long timeSinceLastActive = Time.now() - pool.getLastActiveTime();
-int total = pool.getNumConnections();
-int active = getNumActiveConnections();
-if (timeSinceLastActive > connectionCleanupPeriodMs ||
-active < MIN_ACTIVE_RATIO * total) {
- // Remove and close 1 connection
- List conns = pool.removeConnections(1);
- for (ConnectionContext conn : conns) {
-conn.close();
- }
- LOG.debug("Removed connection {} used {} seconds ago. " +
- "Pool has {}/{} connections", pool.getConnectionPoolId(),
- TimeUnit.MILLISECONDS.toSeconds(timeSinceLastActive),
- pool.getNumConnections(), pool.getMaxSize());
-}
- }
-}
}
/**
http://git-wip-us.apache.org/repos/asf/hadoop/blob/0c2b969e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionPoolId.java
-
hadoop git commit: HDFS-13232. RBF: ConnectionManager's cleanup task will compare each pool's own active conns with its total conns. Contributed by Chao Sun.
Repository: hadoop
Updated Branches:
refs/heads/branch-3.1 ba0184376 -> e27ee2302
HDFS-13232. RBF: ConnectionManager's cleanup task will compare each pool's own
active conns with its total conns. Contributed by Chao Sun.
(cherry picked from commit 0c2b969e0161a068bf9ae013c4b95508dfb90a8a)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/e27ee230
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/e27ee230
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/e27ee230
Branch: refs/heads/branch-3.1
Commit: e27ee2302c2651edfbe04c41a5b38666a83f67d9
Parents: ba01843
Author: Inigo Goiri
Authored: Thu Mar 8 09:32:05 2018 -0800
Committer: Inigo Goiri
Committed: Thu Mar 8 09:32:51 2018 -0800
--
.../federation/router/ConnectionManager.java| 59 +-
.../federation/router/ConnectionPoolId.java | 6 +
.../router/TestConnectionManager.java | 114 +++
3 files changed, 153 insertions(+), 26 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/e27ee230/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java
index 2e45280..594f489 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java
@@ -32,6 +32,7 @@ import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReadWriteLock;
import java.util.concurrent.locks.ReentrantReadWriteLock;
+import com.google.common.annotations.VisibleForTesting;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hdfs.DFSConfigKeys;
import org.apache.hadoop.security.UserGroupInformation;
@@ -303,6 +304,38 @@ public class ConnectionManager {
return JSON.toString(info);
}
+ @VisibleForTesting
+ Map getPools() {
+return this.pools;
+ }
+
+ /**
+ * Clean the unused connections for this pool.
+ *
+ * @param pool Connection pool to cleanup.
+ */
+ @VisibleForTesting
+ void cleanup(ConnectionPool pool) {
+if (pool.getNumConnections() > pool.getMinSize()) {
+ // Check if the pool hasn't been active in a while or not 50% are used
+ long timeSinceLastActive = Time.now() - pool.getLastActiveTime();
+ int total = pool.getNumConnections();
+ int active = pool.getNumActiveConnections();
+ if (timeSinceLastActive > connectionCleanupPeriodMs ||
+ active < MIN_ACTIVE_RATIO * total) {
+// Remove and close 1 connection
+List conns = pool.removeConnections(1);
+for (ConnectionContext conn : conns) {
+ conn.close();
+}
+LOG.debug("Removed connection {} used {} seconds ago. " +
+"Pool has {}/{} connections", pool.getConnectionPoolId(),
+TimeUnit.MILLISECONDS.toSeconds(timeSinceLastActive),
+pool.getNumConnections(), pool.getMaxSize());
+ }
+}
+ }
+
/**
* Removes stale connections not accessed recently from the pool. This is
* invoked periodically.
@@ -350,32 +383,6 @@ public class ConnectionManager {
}
}
}
-
-/**
- * Clean the unused connections for this pool.
- *
- * @param pool Connection pool to cleanup.
- */
-private void cleanup(ConnectionPool pool) {
- if (pool.getNumConnections() > pool.getMinSize()) {
-// Check if the pool hasn't been active in a while or not 50% are used
-long timeSinceLastActive = Time.now() - pool.getLastActiveTime();
-int total = pool.getNumConnections();
-int active = getNumActiveConnections();
-if (timeSinceLastActive > connectionCleanupPeriodMs ||
-active < MIN_ACTIVE_RATIO * total) {
- // Remove and close 1 connection
- List conns = pool.removeConnections(1);
- for (ConnectionContext conn : conns) {
-conn.close();
- }
- LOG.debug("Removed connection {} used {} seconds ago. " +
- "Pool has {}/{} connections", pool.getConnectionPoolId(),
- TimeUnit.MILLISECONDS.toSeconds(timeSinceLastActive),
- pool.getNumConnections(), pool.getMaxSize());
-}
- }
-}
}
/**
http://git-wip-us.apache.org/repos/asf/hadoop/blob/e27ee230/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/ro
hadoop git commit: HDFS-13232. RBF: ConnectionManager's cleanup task will compare each pool's own active conns with its total conns. Contributed by Chao Sun.
Repository: hadoop
Updated Branches:
refs/heads/branch-3.0 7a9aea17a -> 68cff8c23
HDFS-13232. RBF: ConnectionManager's cleanup task will compare each pool's own
active conns with its total conns. Contributed by Chao Sun.
(cherry picked from commit 0c2b969e0161a068bf9ae013c4b95508dfb90a8a)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/68cff8c2
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/68cff8c2
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/68cff8c2
Branch: refs/heads/branch-3.0
Commit: 68cff8c23533ce714edfb9a7f9c1781864e3dc08
Parents: 7a9aea1
Author: Inigo Goiri
Authored: Thu Mar 8 09:32:05 2018 -0800
Committer: Inigo Goiri
Committed: Thu Mar 8 09:33:24 2018 -0800
--
.../federation/router/ConnectionManager.java| 59 +-
.../federation/router/ConnectionPoolId.java | 6 +
.../router/TestConnectionManager.java | 114 +++
3 files changed, 153 insertions(+), 26 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/68cff8c2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java
index 2e45280..594f489 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java
@@ -32,6 +32,7 @@ import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReadWriteLock;
import java.util.concurrent.locks.ReentrantReadWriteLock;
+import com.google.common.annotations.VisibleForTesting;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hdfs.DFSConfigKeys;
import org.apache.hadoop.security.UserGroupInformation;
@@ -303,6 +304,38 @@ public class ConnectionManager {
return JSON.toString(info);
}
+ @VisibleForTesting
+ Map getPools() {
+return this.pools;
+ }
+
+ /**
+ * Clean the unused connections for this pool.
+ *
+ * @param pool Connection pool to cleanup.
+ */
+ @VisibleForTesting
+ void cleanup(ConnectionPool pool) {
+if (pool.getNumConnections() > pool.getMinSize()) {
+ // Check if the pool hasn't been active in a while or not 50% are used
+ long timeSinceLastActive = Time.now() - pool.getLastActiveTime();
+ int total = pool.getNumConnections();
+ int active = pool.getNumActiveConnections();
+ if (timeSinceLastActive > connectionCleanupPeriodMs ||
+ active < MIN_ACTIVE_RATIO * total) {
+// Remove and close 1 connection
+List conns = pool.removeConnections(1);
+for (ConnectionContext conn : conns) {
+ conn.close();
+}
+LOG.debug("Removed connection {} used {} seconds ago. " +
+"Pool has {}/{} connections", pool.getConnectionPoolId(),
+TimeUnit.MILLISECONDS.toSeconds(timeSinceLastActive),
+pool.getNumConnections(), pool.getMaxSize());
+ }
+}
+ }
+
/**
* Removes stale connections not accessed recently from the pool. This is
* invoked periodically.
@@ -350,32 +383,6 @@ public class ConnectionManager {
}
}
}
-
-/**
- * Clean the unused connections for this pool.
- *
- * @param pool Connection pool to cleanup.
- */
-private void cleanup(ConnectionPool pool) {
- if (pool.getNumConnections() > pool.getMinSize()) {
-// Check if the pool hasn't been active in a while or not 50% are used
-long timeSinceLastActive = Time.now() - pool.getLastActiveTime();
-int total = pool.getNumConnections();
-int active = getNumActiveConnections();
-if (timeSinceLastActive > connectionCleanupPeriodMs ||
-active < MIN_ACTIVE_RATIO * total) {
- // Remove and close 1 connection
- List conns = pool.removeConnections(1);
- for (ConnectionContext conn : conns) {
-conn.close();
- }
- LOG.debug("Removed connection {} used {} seconds ago. " +
- "Pool has {}/{} connections", pool.getConnectionPoolId(),
- TimeUnit.MILLISECONDS.toSeconds(timeSinceLastActive),
- pool.getNumConnections(), pool.getMaxSize());
-}
- }
-}
}
/**
http://git-wip-us.apache.org/repos/asf/hadoop/blob/68cff8c2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/ro
hadoop git commit: HDFS-13232. RBF: ConnectionManager's cleanup task will compare each pool's own active conns with its total conns. Contributed by Chao Sun.
Repository: hadoop
Updated Branches:
refs/heads/branch-2.9 e3c80d739 -> 44abf77d9
HDFS-13232. RBF: ConnectionManager's cleanup task will compare each pool's own
active conns with its total conns. Contributed by Chao Sun.
(cherry picked from commit 0c2b969e0161a068bf9ae013c4b95508dfb90a8a)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/44abf77d
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/44abf77d
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/44abf77d
Branch: refs/heads/branch-2.9
Commit: 44abf77d9b371b0d06569bd4881356ac157b490e
Parents: e3c80d7
Author: Inigo Goiri
Authored: Thu Mar 8 09:32:05 2018 -0800
Committer: Inigo Goiri
Committed: Thu Mar 8 09:34:07 2018 -0800
--
.../federation/router/ConnectionManager.java| 59 +-
.../federation/router/ConnectionPoolId.java | 6 +
.../router/TestConnectionManager.java | 114 +++
3 files changed, 153 insertions(+), 26 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/44abf77d/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java
index ebf1556..e94f69b 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java
@@ -32,6 +32,7 @@ import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReadWriteLock;
import java.util.concurrent.locks.ReentrantReadWriteLock;
+import com.google.common.annotations.VisibleForTesting;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hdfs.DFSConfigKeys;
import org.apache.hadoop.security.UserGroupInformation;
@@ -303,6 +304,38 @@ public class ConnectionManager {
return JSON.toString(info);
}
+ @VisibleForTesting
+ Map getPools() {
+return this.pools;
+ }
+
+ /**
+ * Clean the unused connections for this pool.
+ *
+ * @param pool Connection pool to cleanup.
+ */
+ @VisibleForTesting
+ void cleanup(ConnectionPool pool) {
+if (pool.getNumConnections() > pool.getMinSize()) {
+ // Check if the pool hasn't been active in a while or not 50% are used
+ long timeSinceLastActive = Time.now() - pool.getLastActiveTime();
+ int total = pool.getNumConnections();
+ int active = pool.getNumActiveConnections();
+ if (timeSinceLastActive > connectionCleanupPeriodMs ||
+ active < MIN_ACTIVE_RATIO * total) {
+// Remove and close 1 connection
+List conns = pool.removeConnections(1);
+for (ConnectionContext conn : conns) {
+ conn.close();
+}
+LOG.debug("Removed connection {} used {} seconds ago. " +
+"Pool has {}/{} connections", pool.getConnectionPoolId(),
+TimeUnit.MILLISECONDS.toSeconds(timeSinceLastActive),
+pool.getNumConnections(), pool.getMaxSize());
+ }
+}
+ }
+
/**
* Removes stale connections not accessed recently from the pool. This is
* invoked periodically.
@@ -350,32 +383,6 @@ public class ConnectionManager {
}
}
}
-
-/**
- * Clean the unused connections for this pool.
- *
- * @param pool Connection pool to cleanup.
- */
-private void cleanup(ConnectionPool pool) {
- if (pool.getNumConnections() > pool.getMinSize()) {
-// Check if the pool hasn't been active in a while or not 50% are used
-long timeSinceLastActive = Time.now() - pool.getLastActiveTime();
-int total = pool.getNumConnections();
-int active = getNumActiveConnections();
-if (timeSinceLastActive > connectionCleanupPeriodMs ||
-active < MIN_ACTIVE_RATIO * total) {
- // Remove and close 1 connection
- List conns = pool.removeConnections(1);
- for (ConnectionContext conn : conns) {
-conn.close();
- }
- LOG.debug("Removed connection {} used {} seconds ago. " +
- "Pool has {}/{} connections", pool.getConnectionPoolId(),
- TimeUnit.MILLISECONDS.toSeconds(timeSinceLastActive),
- pool.getNumConnections(), pool.getMaxSize());
-}
- }
-}
}
/**
http://git-wip-us.apache.org/repos/asf/hadoop/blob/44abf77d/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/ro
hadoop git commit: HADOOP-15280. TestKMS.testWebHDFSProxyUserKerb and TestKMS.testWebHDFSProxyUserSimple fail in trunk. Contributed by Bharat Viswanadham.
Repository: hadoop
Updated Branches:
refs/heads/trunk 0c2b969e0 -> a906a2264
HADOOP-15280. TestKMS.testWebHDFSProxyUserKerb and
TestKMS.testWebHDFSProxyUserSimple fail in trunk. Contributed by Bharat
Viswanadham.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/a906a226
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/a906a226
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/a906a226
Branch: refs/heads/trunk
Commit: a906a226458a0b4c4b2df61d9bcf375a1d194925
Parents: 0c2b969
Author: Xiao Chen
Authored: Thu Mar 8 10:16:37 2018 -0800
Committer: Xiao Chen
Committed: Thu Mar 8 10:17:02 2018 -0800
--
.../java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java | 6 +-
1 file changed, 5 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/a906a226/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
--
diff --git
a/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
b/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
index f7ecf44..1189fbf 100644
---
a/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
+++
b/hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
@@ -2667,7 +2667,11 @@ public class TestKMS {
kp.createKey("kbb", new KeyProvider.Options(conf));
Assert.fail();
} catch (Exception ex) {
- Assert.assertTrue(ex.getMessage(),
ex.getMessage().contains("Forbidden"));
+ GenericTestUtils.assertExceptionContains("Error while " +
+ "authenticating with endpoint", ex);
+ GenericTestUtils.assertExceptionContains("Forbidden", ex
+ .getCause().getCause());
+
}
return null;
}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: YARN-7944. [UI2] Remove master node link from headers of application pages. Contributed by Yesha Vora.
Repository: hadoop
Updated Branches:
refs/heads/trunk a906a2264 -> 113f401f4
YARN-7944. [UI2] Remove master node link from headers of application pages.
Contributed by Yesha Vora.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/113f401f
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/113f401f
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/113f401f
Branch: refs/heads/trunk
Commit: 113f401f41ee575cb303ceb647bc243108d93a04
Parents: a906a22
Author: Sunil G
Authored: Thu Mar 8 23:52:38 2018 +0530
Committer: Sunil G
Committed: Thu Mar 8 23:53:36 2018 +0530
--
.../src/main/webapp/app/models/yarn-app-timeline.js | 1 -
.../hadoop-yarn-ui/src/main/webapp/app/models/yarn-app.js | 5 -
.../src/main/webapp/app/serializers/yarn-app-timeline.js| 1 -
.../hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-app.js | 1 -
.../hadoop-yarn-ui/src/main/webapp/app/templates/yarn-app.hbs | 2 --
5 files changed, 10 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/113f401f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-app-timeline.js
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-app-timeline.js
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-app-timeline.js
index fa5223f..8b2702f 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-app-timeline.js
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-app-timeline.js
@@ -30,7 +30,6 @@ export default DS.Model.extend({
finishedTime: DS.attr('finishedTime'),
progress: DS.attr('number'),
diagnostics: DS.attr('string'),
- amContainerLogs: DS.attr('string'),
amHostHttpAddress: DS.attr('string'),
logAggregationStatus: DS.attr('string'),
unmanagedApplication: DS.attr('string'),
http://git-wip-us.apache.org/repos/asf/hadoop/blob/113f401f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-app.js
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-app.js
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-app.js
index 5d0f23b..fcc8490 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-app.js
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-app.js
@@ -30,7 +30,6 @@ export default DS.Model.extend({
finishedTime: DS.attr("finishedTime"),
progress: DS.attr("number"),
diagnostics: DS.attr("string"),
- amContainerLogs: DS.attr("string"),
amHostHttpAddress: DS.attr("string"),
masterNodeId: DS.attr("string"),
logAggregationStatus: DS.attr("string"),
@@ -97,10 +96,6 @@ export default DS.Model.extend({
);
}.property("memorySeconds", "vcoreSeconds"),
- masterNodeURL: function() {
-return
`#/yarn-node/${this.get("masterNodeId")}/${this.get("amHostHttpAddress")}/info/`;
- }.property("masterNodeId", "amHostHttpAddress"),
-
progressStyle: function() {
return "width: " + this.get("progress") + "%";
}.property("progress"),
http://git-wip-us.apache.org/repos/asf/hadoop/blob/113f401f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-app-timeline.js
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-app-timeline.js
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-app-timeline.js
index 680fe8c..0496d77 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-app-timeline.js
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-app-timeline.js
@@ -45,7 +45,6 @@ export default DS.JSONAPISerializer.extend({
progress: 100,
applicationType: payload.info.YARN_APPLICATION_TYPE,
diagnostics: (diagnostics && diagnostics !== 'null')? diagnostics : '',
-amContainerLogs: '',
amHostHttpAddress: '',
logAggregationStatus: '',
unmanagedApplication:
payload.info.YARN_APPLICATION_UNMANAGED_APPLICATION || 'N/A',
http://git-wip-us.apache.org/repos/asf/hadoop/blob/113f401f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-app.js
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-app.js
b/hadoop-yarn-pr
hadoop git commit: YARN-7944. [UI2] Remove master node link from headers of application pages. Contributed by Yesha Vora.
Repository: hadoop
Updated Branches:
refs/heads/branch-3.1 e27ee2302 -> f313e5949
YARN-7944. [UI2] Remove master node link from headers of application pages.
Contributed by Yesha Vora.
(cherry picked from commit 113f401f41ee575cb303ceb647bc243108d93a04)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/f313e594
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/f313e594
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/f313e594
Branch: refs/heads/branch-3.1
Commit: f313e594940c5de4f15be9f8da4e8e3e8ee3c81d
Parents: e27ee23
Author: Sunil G
Authored: Thu Mar 8 23:52:38 2018 +0530
Committer: Sunil G
Committed: Thu Mar 8 23:54:15 2018 +0530
--
.../src/main/webapp/app/models/yarn-app-timeline.js | 1 -
.../hadoop-yarn-ui/src/main/webapp/app/models/yarn-app.js | 5 -
.../src/main/webapp/app/serializers/yarn-app-timeline.js| 1 -
.../hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-app.js | 1 -
.../hadoop-yarn-ui/src/main/webapp/app/templates/yarn-app.hbs | 2 --
5 files changed, 10 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/f313e594/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-app-timeline.js
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-app-timeline.js
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-app-timeline.js
index fa5223f..8b2702f 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-app-timeline.js
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-app-timeline.js
@@ -30,7 +30,6 @@ export default DS.Model.extend({
finishedTime: DS.attr('finishedTime'),
progress: DS.attr('number'),
diagnostics: DS.attr('string'),
- amContainerLogs: DS.attr('string'),
amHostHttpAddress: DS.attr('string'),
logAggregationStatus: DS.attr('string'),
unmanagedApplication: DS.attr('string'),
http://git-wip-us.apache.org/repos/asf/hadoop/blob/f313e594/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-app.js
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-app.js
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-app.js
index 5d0f23b..fcc8490 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-app.js
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-app.js
@@ -30,7 +30,6 @@ export default DS.Model.extend({
finishedTime: DS.attr("finishedTime"),
progress: DS.attr("number"),
diagnostics: DS.attr("string"),
- amContainerLogs: DS.attr("string"),
amHostHttpAddress: DS.attr("string"),
masterNodeId: DS.attr("string"),
logAggregationStatus: DS.attr("string"),
@@ -97,10 +96,6 @@ export default DS.Model.extend({
);
}.property("memorySeconds", "vcoreSeconds"),
- masterNodeURL: function() {
-return
`#/yarn-node/${this.get("masterNodeId")}/${this.get("amHostHttpAddress")}/info/`;
- }.property("masterNodeId", "amHostHttpAddress"),
-
progressStyle: function() {
return "width: " + this.get("progress") + "%";
}.property("progress"),
http://git-wip-us.apache.org/repos/asf/hadoop/blob/f313e594/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-app-timeline.js
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-app-timeline.js
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-app-timeline.js
index 680fe8c..0496d77 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-app-timeline.js
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-app-timeline.js
@@ -45,7 +45,6 @@ export default DS.JSONAPISerializer.extend({
progress: 100,
applicationType: payload.info.YARN_APPLICATION_TYPE,
diagnostics: (diagnostics && diagnostics !== 'null')? diagnostics : '',
-amContainerLogs: '',
amHostHttpAddress: '',
logAggregationStatus: '',
unmanagedApplication:
payload.info.YARN_APPLICATION_UNMANAGED_APPLICATION || 'N/A',
http://git-wip-us.apache.org/repos/asf/hadoop/blob/f313e594/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-app.js
--
diff --git
a/hadoop-yarn-project/hadoop-ya
hadoop git commit: HDFS-13232. RBF: ConnectionManager's cleanup task will compare each pool's own active conns with its total conns. Contributed by Chao Sun.
Repository: hadoop
Updated Branches:
refs/heads/branch-2 207daabbc -> 95f34ce46
HDFS-13232. RBF: ConnectionManager's cleanup task will compare each pool's own
active conns with its total conns. Contributed by Chao Sun.
(cherry picked from commit 0c2b969e0161a068bf9ae013c4b95508dfb90a8a)
(cherry picked from commit 44abf77d9b371b0d06569bd4881356ac157b490e)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/95f34ce4
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/95f34ce4
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/95f34ce4
Branch: refs/heads/branch-2
Commit: 95f34ce462b23b632147a9877c9904c11832afce
Parents: 207daab
Author: Inigo Goiri
Authored: Thu Mar 8 09:32:05 2018 -0800
Committer: Inigo Goiri
Committed: Thu Mar 8 10:24:41 2018 -0800
--
.../federation/router/ConnectionManager.java| 59 +-
.../federation/router/ConnectionPoolId.java | 6 +
.../router/TestConnectionManager.java | 114 +++
3 files changed, 153 insertions(+), 26 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/95f34ce4/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java
index ebf1556..e94f69b 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java
@@ -32,6 +32,7 @@ import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReadWriteLock;
import java.util.concurrent.locks.ReentrantReadWriteLock;
+import com.google.common.annotations.VisibleForTesting;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hdfs.DFSConfigKeys;
import org.apache.hadoop.security.UserGroupInformation;
@@ -303,6 +304,38 @@ public class ConnectionManager {
return JSON.toString(info);
}
+ @VisibleForTesting
+ Map getPools() {
+return this.pools;
+ }
+
+ /**
+ * Clean the unused connections for this pool.
+ *
+ * @param pool Connection pool to cleanup.
+ */
+ @VisibleForTesting
+ void cleanup(ConnectionPool pool) {
+if (pool.getNumConnections() > pool.getMinSize()) {
+ // Check if the pool hasn't been active in a while or not 50% are used
+ long timeSinceLastActive = Time.now() - pool.getLastActiveTime();
+ int total = pool.getNumConnections();
+ int active = pool.getNumActiveConnections();
+ if (timeSinceLastActive > connectionCleanupPeriodMs ||
+ active < MIN_ACTIVE_RATIO * total) {
+// Remove and close 1 connection
+List conns = pool.removeConnections(1);
+for (ConnectionContext conn : conns) {
+ conn.close();
+}
+LOG.debug("Removed connection {} used {} seconds ago. " +
+"Pool has {}/{} connections", pool.getConnectionPoolId(),
+TimeUnit.MILLISECONDS.toSeconds(timeSinceLastActive),
+pool.getNumConnections(), pool.getMaxSize());
+ }
+}
+ }
+
/**
* Removes stale connections not accessed recently from the pool. This is
* invoked periodically.
@@ -350,32 +383,6 @@ public class ConnectionManager {
}
}
}
-
-/**
- * Clean the unused connections for this pool.
- *
- * @param pool Connection pool to cleanup.
- */
-private void cleanup(ConnectionPool pool) {
- if (pool.getNumConnections() > pool.getMinSize()) {
-// Check if the pool hasn't been active in a while or not 50% are used
-long timeSinceLastActive = Time.now() - pool.getLastActiveTime();
-int total = pool.getNumConnections();
-int active = getNumActiveConnections();
-if (timeSinceLastActive > connectionCleanupPeriodMs ||
-active < MIN_ACTIVE_RATIO * total) {
- // Remove and close 1 connection
- List conns = pool.removeConnections(1);
- for (ConnectionContext conn : conns) {
-conn.close();
- }
- LOG.debug("Removed connection {} used {} seconds ago. " +
- "Pool has {}/{} connections", pool.getConnectionPoolId(),
- TimeUnit.MILLISECONDS.toSeconds(timeSinceLastActive),
- pool.getNumConnections(), pool.getMaxSize());
-}
- }
-}
}
/**
http://git-wip-us.apache.org/repos/asf/hadoop/blob/95f34ce4/hadoop-hdfs-project/hado
hadoop git commit: HDFS-12614. FSPermissionChecker#getINodeAttrs() throws NPE when INodeAttributesProvider configured. Contributed by Manoj Govindassamy.
Repository: hadoop
Updated Branches:
refs/heads/branch-2 95f34ce46 -> 0aa52d408
HDFS-12614. FSPermissionChecker#getINodeAttrs() throws NPE when
INodeAttributesProvider configured. Contributed by Manoj Govindassamy.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/0aa52d40
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/0aa52d40
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/0aa52d40
Branch: refs/heads/branch-2
Commit: 0aa52d4085f8c77dbfd9d913011c347882573ad9
Parents: 95f34ce
Author: Kihwal Lee
Authored: Thu Mar 8 14:20:09 2018 -0600
Committer: Kihwal Lee
Committed: Thu Mar 8 14:22:17 2018 -0600
--
.../server/namenode/FSPermissionChecker.java| 12 +++-
.../namenode/TestINodeAttributeProvider.java| 63 +++-
2 files changed, 57 insertions(+), 18 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/0aa52d40/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSPermissionChecker.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSPermissionChecker.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSPermissionChecker.java
index 22f9b99..8c53308 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSPermissionChecker.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSPermissionChecker.java
@@ -271,8 +271,16 @@ public class FSPermissionChecker implements
AccessControlEnforcer {
INodeAttributes inodeAttrs = inode.getSnapshotINode(snapshotId);
if (getAttributesProvider() != null) {
String[] elements = new String[pathIdx + 1];
- for (int i = 0; i < elements.length; i++) {
-elements[i] = DFSUtil.bytes2String(pathByNameArr[i]);
+ /**
+ * {@link INode#getPathComponents(String)} returns a null component
+ * for the root only path "/". Assign an empty string if so.
+ */
+ if (pathByNameArr.length == 1 && pathByNameArr[0] == null) {
+elements[0] = "";
+ } else {
+for (int i = 0; i < elements.length; i++) {
+ elements[i] = DFSUtil.bytes2String(pathByNameArr[i]);
+}
}
inodeAttrs = getAttributesProvider().getAttributes(elements, inodeAttrs);
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/0aa52d40/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeAttributeProvider.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeAttributeProvider.java
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeAttributeProvider.java
index bbc5fa0..5495692 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeAttributeProvider.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeAttributeProvider.java
@@ -313,31 +313,62 @@ public class TestINodeAttributeProvider {
testBypassProviderHelper(users, HDFS_PERMISSION, true);
}
- @Test
- public void testCustomProvider() throws Exception {
+ private void verifyFileStatus(UserGroupInformation ugi) throws IOException {
FileSystem fs = FileSystem.get(miniDFS.getConfiguration(0));
-fs.mkdirs(new Path("/user/xxx"));
-FileStatus status = fs.getFileStatus(new Path("/user/xxx"));
-Assert.assertEquals(System.getProperty("user.name"), status.getOwner());
+
+FileStatus status = fs.getFileStatus(new Path("/"));
+LOG.info("Path '/' is owned by: "
++ status.getOwner() + ":" + status.getGroup());
+
+Path userDir = new Path("/user/" + ugi.getShortUserName());
+fs.mkdirs(userDir);
+status = fs.getFileStatus(userDir);
+Assert.assertEquals(ugi.getShortUserName(), status.getOwner());
Assert.assertEquals("supergroup", status.getGroup());
Assert.assertEquals(new FsPermission((short) 0755),
status.getPermission());
-fs.mkdirs(new Path("/user/authz"));
-Path p = new Path("/user/authz");
-status = fs.getFileStatus(p);
+
+Path authzDir = new Path("/user/authz");
+fs.mkdirs(authzDir);
+status = fs.getFileStatus(authzDir);
Assert.assertEquals("foo", status.getOwner());
Assert.assertEquals("bar", status.getGroup());
Assert.assertEquals(new FsPermission((short) 0770),
status.getPermission());
-AclStatus aclStatus = fs.getAclStatus(p);
+
+AclStatus aclStatus = fs.getAclStatus(authzDir);
Assert.assertEquals(
hadoop git commit: HDFS-12614. FSPermissionChecker#getINodeAttrs() throws NPE when INodeAttributesProvider configured. Contributed by Manoj Govindassamy.
Repository: hadoop
Updated Branches:
refs/heads/branch-2.9 44abf77d9 -> 6fafaa7b7
HDFS-12614. FSPermissionChecker#getINodeAttrs() throws NPE when
INodeAttributesProvider configured. Contributed by Manoj Govindassamy.
(cherry picked from commit 0aa52d4085f8c77dbfd9d913011c347882573ad9)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/6fafaa7b
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/6fafaa7b
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/6fafaa7b
Branch: refs/heads/branch-2.9
Commit: 6fafaa7b7fafa2005386779fb4c92750f256c336
Parents: 44abf77
Author: Kihwal Lee
Authored: Thu Mar 8 14:26:42 2018 -0600
Committer: Kihwal Lee
Committed: Thu Mar 8 14:26:42 2018 -0600
--
.../server/namenode/FSPermissionChecker.java| 12 +++-
.../namenode/TestINodeAttributeProvider.java| 63 +++-
2 files changed, 57 insertions(+), 18 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/6fafaa7b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSPermissionChecker.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSPermissionChecker.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSPermissionChecker.java
index 388e91f..ec82ab7 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSPermissionChecker.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSPermissionChecker.java
@@ -271,8 +271,16 @@ public class FSPermissionChecker implements
AccessControlEnforcer {
INodeAttributes inodeAttrs = inode.getSnapshotINode(snapshotId);
if (getAttributesProvider() != null) {
String[] elements = new String[pathIdx + 1];
- for (int i = 0; i < elements.length; i++) {
-elements[i] = DFSUtil.bytes2String(pathByNameArr[i]);
+ /**
+ * {@link INode#getPathComponents(String)} returns a null component
+ * for the root only path "/". Assign an empty string if so.
+ */
+ if (pathByNameArr.length == 1 && pathByNameArr[0] == null) {
+elements[0] = "";
+ } else {
+for (int i = 0; i < elements.length; i++) {
+ elements[i] = DFSUtil.bytes2String(pathByNameArr[i]);
+}
}
inodeAttrs = getAttributesProvider().getAttributes(elements, inodeAttrs);
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/6fafaa7b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeAttributeProvider.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeAttributeProvider.java
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeAttributeProvider.java
index bbc5fa0..5495692 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeAttributeProvider.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeAttributeProvider.java
@@ -313,31 +313,62 @@ public class TestINodeAttributeProvider {
testBypassProviderHelper(users, HDFS_PERMISSION, true);
}
- @Test
- public void testCustomProvider() throws Exception {
+ private void verifyFileStatus(UserGroupInformation ugi) throws IOException {
FileSystem fs = FileSystem.get(miniDFS.getConfiguration(0));
-fs.mkdirs(new Path("/user/xxx"));
-FileStatus status = fs.getFileStatus(new Path("/user/xxx"));
-Assert.assertEquals(System.getProperty("user.name"), status.getOwner());
+
+FileStatus status = fs.getFileStatus(new Path("/"));
+LOG.info("Path '/' is owned by: "
++ status.getOwner() + ":" + status.getGroup());
+
+Path userDir = new Path("/user/" + ugi.getShortUserName());
+fs.mkdirs(userDir);
+status = fs.getFileStatus(userDir);
+Assert.assertEquals(ugi.getShortUserName(), status.getOwner());
Assert.assertEquals("supergroup", status.getGroup());
Assert.assertEquals(new FsPermission((short) 0755),
status.getPermission());
-fs.mkdirs(new Path("/user/authz"));
-Path p = new Path("/user/authz");
-status = fs.getFileStatus(p);
+
+Path authzDir = new Path("/user/authz");
+fs.mkdirs(authzDir);
+status = fs.getFileStatus(authzDir);
Assert.assertEquals("foo", status.getOwner());
Assert.assertEquals("bar", status.getGroup());
Assert.assertEquals(new FsPermission((short) 0770),
status.getPermission());
-AclStatus aclStatus = fs.getAclStatus(p);
+
+
hadoop git commit: HDFS-12614. FSPermissionChecker#getINodeAttrs() throws NPE when INodeAttributesProvider configured. Contributed by Manoj Govindassamy.
Repository: hadoop
Updated Branches:
refs/heads/branch-2.8 4b0b466f8 -> f4621e022
HDFS-12614. FSPermissionChecker#getINodeAttrs() throws NPE when
INodeAttributesProvider configured. Contributed by Manoj Govindassamy.
(cherry picked from commit 0aa52d4085f8c77dbfd9d913011c347882573ad9)
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeAttributeProvider.java
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/f4621e02
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/f4621e02
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/f4621e02
Branch: refs/heads/branch-2.8
Commit: f4621e0221ef14291ac616dc43c673b50a6d1550
Parents: 4b0b466
Author: Kihwal Lee
Authored: Thu Mar 8 14:38:47 2018 -0600
Committer: Kihwal Lee
Committed: Thu Mar 8 14:38:47 2018 -0600
--
.../server/namenode/FSPermissionChecker.java| 12 +++-
.../namenode/TestINodeAttributeProvider.java| 60 ++--
2 files changed, 54 insertions(+), 18 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/f4621e02/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSPermissionChecker.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSPermissionChecker.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSPermissionChecker.java
index 107d563..46d0959 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSPermissionChecker.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSPermissionChecker.java
@@ -239,8 +239,16 @@ class FSPermissionChecker implements AccessControlEnforcer
{
INodeAttributes inodeAttrs = inode.getSnapshotINode(snapshotId);
if (getAttributesProvider() != null) {
String[] elements = new String[pathIdx + 1];
- for (int i = 0; i < elements.length; i++) {
-elements[i] = DFSUtil.bytes2String(pathByNameArr[i]);
+ /**
+ * {@link INode#getPathComponents(String)} returns a null component
+ * for the root only path "/". Assign an empty string if so.
+ */
+ if (pathByNameArr.length == 1 && pathByNameArr[0] == null) {
+elements[0] = "";
+ } else {
+for (int i = 0; i < elements.length; i++) {
+ elements[i] = DFSUtil.bytes2String(pathByNameArr[i]);
+}
}
inodeAttrs = getAttributesProvider().getAttributes(elements, inodeAttrs);
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/f4621e02/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeAttributeProvider.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeAttributeProvider.java
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeAttributeProvider.java
index ffdc535..1837525 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeAttributeProvider.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeAttributeProvider.java
@@ -222,31 +222,59 @@ public class TestINodeAttributeProvider {
});
}
- @Test
- public void testCustomProvider() throws Exception {
+ private void verifyFileStatus(UserGroupInformation ugi) throws IOException {
FileSystem fs = FileSystem.get(miniDFS.getConfiguration(0));
-fs.mkdirs(new Path("/user/xxx"));
-FileStatus status = fs.getFileStatus(new Path("/user/xxx"));
-Assert.assertEquals(System.getProperty("user.name"), status.getOwner());
+
+FileStatus status = fs.getFileStatus(new Path("/"));
+Path userDir = new Path("/user/" + ugi.getShortUserName());
+fs.mkdirs(userDir);
+status = fs.getFileStatus(userDir);
+Assert.assertEquals(ugi.getShortUserName(), status.getOwner());
Assert.assertEquals("supergroup", status.getGroup());
Assert.assertEquals(new FsPermission((short) 0755),
status.getPermission());
-fs.mkdirs(new Path("/user/authz"));
-Path p = new Path("/user/authz");
-status = fs.getFileStatus(p);
+
+Path authzDir = new Path("/user/authz");
+fs.mkdirs(authzDir);
+status = fs.getFileStatus(authzDir);
Assert.assertEquals("foo", status.getOwner());
Assert.assertEquals("bar", status.getGroup());
Assert.assertEquals(new FsPermission((short) 0770),
status.getPermission());
-AclStatus aclStatus = fs.getAclStatus(p);
+
+AclStatus aclStatus
hadoop git commit: HDFS-13233. RBF: MountTableResolver doesn't return the correct mount point of the given path. Contributed by wangzhiyuan.
Repository: hadoop
Updated Branches:
refs/heads/trunk 113f401f4 -> 122805b43
HDFS-13233. RBF: MountTableResolver doesn't return the correct mount point of
the given path. Contributed by wangzhiyuan.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/122805b4
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/122805b4
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/122805b4
Branch: refs/heads/trunk
Commit: 122805b43acff2b094bd984fa76dbc8d2e110edd
Parents: 113f401
Author: Yiqun Lin
Authored: Fri Mar 9 15:42:57 2018 +0800
Committer: Yiqun Lin
Committed: Fri Mar 9 15:42:57 2018 +0800
--
.../federation/resolver/MountTableResolver.java | 13 ++-
.../resolver/TestMountTableResolver.java| 23
.../federation/router/TestRouterQuota.java | 2 +-
3 files changed, 36 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/122805b4/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java
index 374e3ba..dac6f7f 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java
@@ -521,6 +521,17 @@ public class MountTableResolver
return this.defaultNameService;
}
+ private boolean isParentEntry(final String path, final String parent) {
+if (!path.startsWith(parent)) {
+ return false;
+}
+if (path.equals(parent)) {
+ return true;
+}
+return path.charAt(parent.length()) == Path.SEPARATOR_CHAR
+|| parent.equals(Path.SEPARATOR);
+ }
+
/**
* Find the deepest mount point for a path.
* @param path Path to look for.
@@ -530,7 +541,7 @@ public class MountTableResolver
readLock.lock();
try {
Entry entry = this.tree.floorEntry(path);
- while (entry != null && !path.startsWith(entry.getKey())) {
+ while (entry != null && !isParentEntry(path, entry.getKey())) {
entry = this.tree.lowerEntry(entry.getKey());
}
if (entry == null) {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/122805b4/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestMountTableResolver.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestMountTableResolver.java
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestMountTableResolver.java
index fa2f89c..a09daf0 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestMountTableResolver.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestMountTableResolver.java
@@ -179,6 +179,29 @@ public class TestMountTableResolver {
}
@Test
+ public void testGetMountPoint() throws IOException {
+// Check get the mount table entry for a path
+MountTable mtEntry;
+mtEntry = mountTable.getMountPoint("/");
+assertTrue(mtEntry.getSourcePath().equals("/"));
+
+mtEntry = mountTable.getMountPoint("/user");
+assertTrue(mtEntry.getSourcePath().equals("/user"));
+
+mtEntry = mountTable.getMountPoint("/user/a");
+assertTrue(mtEntry.getSourcePath().equals("/user/a"));
+
+mtEntry = mountTable.getMountPoint("/user/a/");
+assertTrue(mtEntry.getSourcePath().equals("/user/a"));
+
+mtEntry = mountTable.getMountPoint("/user/a/11");
+assertTrue(mtEntry.getSourcePath().equals("/user/a"));
+
+mtEntry = mountTable.getMountPoint("/user/a1");
+assertTrue(mtEntry.getSourcePath().equals("/user"));
+ }
+
+ @Test
public void testGetMountPoints() throws IOException {
// Check getting all mount points (virtual and real) beneath a path
http://git-wip-us.apache.org/repos/asf/hadoop/blob/122805b4/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterQuota.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterQuota.java
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/or
hadoop git commit: HDFS-13233. RBF: MountTableResolver doesn't return the correct mount point of the given path. Contributed by wangzhiyuan.
Repository: hadoop
Updated Branches:
refs/heads/branch-3.1 f313e5949 -> 9368f7fba
HDFS-13233. RBF: MountTableResolver doesn't return the correct mount point of
the given path. Contributed by wangzhiyuan.
(cherry picked from commit 122805b43acff2b094bd984fa76dbc8d2e110edd)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/9368f7fb
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/9368f7fb
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/9368f7fb
Branch: refs/heads/branch-3.1
Commit: 9368f7fbabaf4b60624cee368db8a80ba58545b4
Parents: f313e59
Author: Yiqun Lin
Authored: Fri Mar 9 15:42:57 2018 +0800
Committer: Yiqun Lin
Committed: Fri Mar 9 15:44:35 2018 +0800
--
.../federation/resolver/MountTableResolver.java | 13 ++-
.../resolver/TestMountTableResolver.java| 23
.../federation/router/TestRouterQuota.java | 2 +-
3 files changed, 36 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9368f7fb/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java
index 374e3ba..dac6f7f 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java
@@ -521,6 +521,17 @@ public class MountTableResolver
return this.defaultNameService;
}
+ private boolean isParentEntry(final String path, final String parent) {
+if (!path.startsWith(parent)) {
+ return false;
+}
+if (path.equals(parent)) {
+ return true;
+}
+return path.charAt(parent.length()) == Path.SEPARATOR_CHAR
+|| parent.equals(Path.SEPARATOR);
+ }
+
/**
* Find the deepest mount point for a path.
* @param path Path to look for.
@@ -530,7 +541,7 @@ public class MountTableResolver
readLock.lock();
try {
Entry entry = this.tree.floorEntry(path);
- while (entry != null && !path.startsWith(entry.getKey())) {
+ while (entry != null && !isParentEntry(path, entry.getKey())) {
entry = this.tree.lowerEntry(entry.getKey());
}
if (entry == null) {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9368f7fb/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestMountTableResolver.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestMountTableResolver.java
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestMountTableResolver.java
index fa2f89c..a09daf0 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestMountTableResolver.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestMountTableResolver.java
@@ -179,6 +179,29 @@ public class TestMountTableResolver {
}
@Test
+ public void testGetMountPoint() throws IOException {
+// Check get the mount table entry for a path
+MountTable mtEntry;
+mtEntry = mountTable.getMountPoint("/");
+assertTrue(mtEntry.getSourcePath().equals("/"));
+
+mtEntry = mountTable.getMountPoint("/user");
+assertTrue(mtEntry.getSourcePath().equals("/user"));
+
+mtEntry = mountTable.getMountPoint("/user/a");
+assertTrue(mtEntry.getSourcePath().equals("/user/a"));
+
+mtEntry = mountTable.getMountPoint("/user/a/");
+assertTrue(mtEntry.getSourcePath().equals("/user/a"));
+
+mtEntry = mountTable.getMountPoint("/user/a/11");
+assertTrue(mtEntry.getSourcePath().equals("/user/a"));
+
+mtEntry = mountTable.getMountPoint("/user/a1");
+assertTrue(mtEntry.getSourcePath().equals("/user"));
+ }
+
+ @Test
public void testGetMountPoints() throws IOException {
// Check getting all mount points (virtual and real) beneath a path
http://git-wip-us.apache.org/repos/asf/hadoop/blob/9368f7fb/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterQuota.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/
hadoop git commit: HDFS-13233. RBF: MountTableResolver doesn't return the correct mount point of the given path. Contributed by wangzhiyuan.
Repository: hadoop
Updated Branches:
refs/heads/branch-3.0 68cff8c23 -> f879504fe
HDFS-13233. RBF: MountTableResolver doesn't return the correct mount point of
the given path. Contributed by wangzhiyuan.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/f879504f
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/f879504f
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/f879504f
Branch: refs/heads/branch-3.0
Commit: f879504fe13267690e4306160210536c1b21b8e3
Parents: 68cff8c
Author: Yiqun Lin
Authored: Fri Mar 9 15:48:36 2018 +0800
Committer: Yiqun Lin
Committed: Fri Mar 9 15:48:36 2018 +0800
--
.../federation/resolver/MountTableResolver.java | 13 ++-
.../resolver/TestMountTableResolver.java| 23
2 files changed, 35 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/f879504f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java
index 374e3ba..dac6f7f 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java
@@ -521,6 +521,17 @@ public class MountTableResolver
return this.defaultNameService;
}
+ private boolean isParentEntry(final String path, final String parent) {
+if (!path.startsWith(parent)) {
+ return false;
+}
+if (path.equals(parent)) {
+ return true;
+}
+return path.charAt(parent.length()) == Path.SEPARATOR_CHAR
+|| parent.equals(Path.SEPARATOR);
+ }
+
/**
* Find the deepest mount point for a path.
* @param path Path to look for.
@@ -530,7 +541,7 @@ public class MountTableResolver
readLock.lock();
try {
Entry entry = this.tree.floorEntry(path);
- while (entry != null && !path.startsWith(entry.getKey())) {
+ while (entry != null && !isParentEntry(path, entry.getKey())) {
entry = this.tree.lowerEntry(entry.getKey());
}
if (entry == null) {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/f879504f/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestMountTableResolver.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestMountTableResolver.java
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestMountTableResolver.java
index fa2f89c..a09daf0 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestMountTableResolver.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestMountTableResolver.java
@@ -179,6 +179,29 @@ public class TestMountTableResolver {
}
@Test
+ public void testGetMountPoint() throws IOException {
+// Check get the mount table entry for a path
+MountTable mtEntry;
+mtEntry = mountTable.getMountPoint("/");
+assertTrue(mtEntry.getSourcePath().equals("/"));
+
+mtEntry = mountTable.getMountPoint("/user");
+assertTrue(mtEntry.getSourcePath().equals("/user"));
+
+mtEntry = mountTable.getMountPoint("/user/a");
+assertTrue(mtEntry.getSourcePath().equals("/user/a"));
+
+mtEntry = mountTable.getMountPoint("/user/a/");
+assertTrue(mtEntry.getSourcePath().equals("/user/a"));
+
+mtEntry = mountTable.getMountPoint("/user/a/11");
+assertTrue(mtEntry.getSourcePath().equals("/user/a"));
+
+mtEntry = mountTable.getMountPoint("/user/a1");
+assertTrue(mtEntry.getSourcePath().equals("/user"));
+ }
+
+ @Test
public void testGetMountPoints() throws IOException {
// Check getting all mount points (virtual and real) beneath a path
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HDFS-13233. RBF: MountTableResolver doesn't return the correct mount point of the given path. Contributed by wangzhiyuan.
Repository: hadoop
Updated Branches:
refs/heads/branch-2 0aa52d408 -> aa748c6bc
HDFS-13233. RBF: MountTableResolver doesn't return the correct mount point of
the given path. Contributed by wangzhiyuan.
(cherry picked from commit 122805b43acff2b094bd984fa76dbc8d2e110edd)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/aa748c6b
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/aa748c6b
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/aa748c6b
Branch: refs/heads/branch-2
Commit: aa748c6bca6ee75eec24c27f7fb32eda01b720b6
Parents: 0aa52d4
Author: Yiqun Lin
Authored: Fri Mar 9 15:42:57 2018 +0800
Committer: Yiqun Lin
Committed: Fri Mar 9 15:51:56 2018 +0800
--
.../federation/resolver/MountTableResolver.java | 13 ++-
.../resolver/TestMountTableResolver.java| 23
.../federation/router/TestRouterQuota.java | 2 +-
3 files changed, 36 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/aa748c6b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java
index 374e3ba..dac6f7f 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java
@@ -521,6 +521,17 @@ public class MountTableResolver
return this.defaultNameService;
}
+ private boolean isParentEntry(final String path, final String parent) {
+if (!path.startsWith(parent)) {
+ return false;
+}
+if (path.equals(parent)) {
+ return true;
+}
+return path.charAt(parent.length()) == Path.SEPARATOR_CHAR
+|| parent.equals(Path.SEPARATOR);
+ }
+
/**
* Find the deepest mount point for a path.
* @param path Path to look for.
@@ -530,7 +541,7 @@ public class MountTableResolver
readLock.lock();
try {
Entry entry = this.tree.floorEntry(path);
- while (entry != null && !path.startsWith(entry.getKey())) {
+ while (entry != null && !isParentEntry(path, entry.getKey())) {
entry = this.tree.lowerEntry(entry.getKey());
}
if (entry == null) {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/aa748c6b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestMountTableResolver.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestMountTableResolver.java
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestMountTableResolver.java
index fa2f89c..a09daf0 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestMountTableResolver.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestMountTableResolver.java
@@ -179,6 +179,29 @@ public class TestMountTableResolver {
}
@Test
+ public void testGetMountPoint() throws IOException {
+// Check get the mount table entry for a path
+MountTable mtEntry;
+mtEntry = mountTable.getMountPoint("/");
+assertTrue(mtEntry.getSourcePath().equals("/"));
+
+mtEntry = mountTable.getMountPoint("/user");
+assertTrue(mtEntry.getSourcePath().equals("/user"));
+
+mtEntry = mountTable.getMountPoint("/user/a");
+assertTrue(mtEntry.getSourcePath().equals("/user/a"));
+
+mtEntry = mountTable.getMountPoint("/user/a/");
+assertTrue(mtEntry.getSourcePath().equals("/user/a"));
+
+mtEntry = mountTable.getMountPoint("/user/a/11");
+assertTrue(mtEntry.getSourcePath().equals("/user/a"));
+
+mtEntry = mountTable.getMountPoint("/user/a1");
+assertTrue(mtEntry.getSourcePath().equals("/user"));
+ }
+
+ @Test
public void testGetMountPoints() throws IOException {
// Check getting all mount points (virtual and real) beneath a path
http://git-wip-us.apache.org/repos/asf/hadoop/blob/aa748c6b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterQuota.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/rout
hadoop git commit: HDFS-13233. RBF: MountTableResolver doesn't return the correct mount point of the given path. Contributed by wangzhiyuan.
Repository: hadoop
Updated Branches:
refs/heads/branch-2.9 6fafaa7b7 -> 04e18e747
HDFS-13233. RBF: MountTableResolver doesn't return the correct mount point of
the given path. Contributed by wangzhiyuan.
(cherry picked from commit f879504fe13267690e4306160210536c1b21b8e3)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/04e18e74
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/04e18e74
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/04e18e74
Branch: refs/heads/branch-2.9
Commit: 04e18e747b3bef9477c0897822101246029dfa2a
Parents: 6fafaa7
Author: Yiqun Lin
Authored: Fri Mar 9 15:48:36 2018 +0800
Committer: Yiqun Lin
Committed: Fri Mar 9 15:53:17 2018 +0800
--
.../federation/resolver/MountTableResolver.java | 13 ++-
.../resolver/TestMountTableResolver.java| 23
2 files changed, 35 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/04e18e74/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java
index 374e3ba..dac6f7f 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/MountTableResolver.java
@@ -521,6 +521,17 @@ public class MountTableResolver
return this.defaultNameService;
}
+ private boolean isParentEntry(final String path, final String parent) {
+if (!path.startsWith(parent)) {
+ return false;
+}
+if (path.equals(parent)) {
+ return true;
+}
+return path.charAt(parent.length()) == Path.SEPARATOR_CHAR
+|| parent.equals(Path.SEPARATOR);
+ }
+
/**
* Find the deepest mount point for a path.
* @param path Path to look for.
@@ -530,7 +541,7 @@ public class MountTableResolver
readLock.lock();
try {
Entry entry = this.tree.floorEntry(path);
- while (entry != null && !path.startsWith(entry.getKey())) {
+ while (entry != null && !isParentEntry(path, entry.getKey())) {
entry = this.tree.lowerEntry(entry.getKey());
}
if (entry == null) {
http://git-wip-us.apache.org/repos/asf/hadoop/blob/04e18e74/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestMountTableResolver.java
--
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestMountTableResolver.java
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestMountTableResolver.java
index fa2f89c..a09daf0 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestMountTableResolver.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/resolver/TestMountTableResolver.java
@@ -179,6 +179,29 @@ public class TestMountTableResolver {
}
@Test
+ public void testGetMountPoint() throws IOException {
+// Check get the mount table entry for a path
+MountTable mtEntry;
+mtEntry = mountTable.getMountPoint("/");
+assertTrue(mtEntry.getSourcePath().equals("/"));
+
+mtEntry = mountTable.getMountPoint("/user");
+assertTrue(mtEntry.getSourcePath().equals("/user"));
+
+mtEntry = mountTable.getMountPoint("/user/a");
+assertTrue(mtEntry.getSourcePath().equals("/user/a"));
+
+mtEntry = mountTable.getMountPoint("/user/a/");
+assertTrue(mtEntry.getSourcePath().equals("/user/a"));
+
+mtEntry = mountTable.getMountPoint("/user/a/11");
+assertTrue(mtEntry.getSourcePath().equals("/user/a"));
+
+mtEntry = mountTable.getMountPoint("/user/a1");
+assertTrue(mtEntry.getSourcePath().equals("/user"));
+ }
+
+ @Test
public void testGetMountPoints() throws IOException {
// Check getting all mount points (virtual and real) beneath a path
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
