[GitHub] [hadoop] tomscut commented on a change in pull request #3062: HDFS-16048. RBF: Print network topology on the router web
tomscut commented on a change in pull request #3062: URL: https://github.com/apache/hadoop/pull/3062#discussion_r644513026 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NetworkTopologyServlet.java ## @@ -90,7 +90,7 @@ public void doGet(HttpServletRequest request, HttpServletResponse response) * @param leaves leaves nodes under base scope * @param format the response format */ - public void printTopology(PrintStream stream, List leaves, + private void printTopology(PrintStream stream, List leaves, Review comment: > Why not just override? Because the arguments of the two methods are a little different, I'll fix it. Thank you for pointing that out. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17590) ABFS: Introduce Lease Operations with Append to provide single writer semantics
[
https://issues.apache.org/jira/browse/HADOOP-17590?focusedWorklogId=605675&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605675
]
ASF GitHub Bot logged work on HADOOP-17590:
---
Author: ASF GitHub Bot
Created on: 03/Jun/21 06:01
Start Date: 03/Jun/21 06:01
Worklog Time Spent: 10m
Work Description: snvijaya commented on a change in pull request #3026:
URL: https://github.com/apache/hadoop/pull/3026#discussion_r643842160
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##
@@ -1698,14 +1703,29 @@ private void updateInfiniteLeaseDirs() {
this.azureInfiniteLeaseDirSet.remove("");
}
- private AbfsLease maybeCreateLease(String relativePath)
+ private AbfsLease maybeCreateFiniteLease(String relativePath, boolean
isNamespaceEnabled)
throws AzureBlobFileSystemException {
boolean enableInfiniteLease = isInfiniteLeaseKey(relativePath);
-if (!enableInfiniteLease) {
- return null;
+AbfsLease lease = null;
+if (!enableInfiniteLease && abfsConfiguration.isLeaseEnforced() &&
isNamespaceEnabled) {
+ lease = new AbfsLease(client, relativePath, false);
Review comment:
Are there scenarios where a finite lease created needs to explicitly
call on acquireLease API ? If that dependency isnt there, it would be better to
create 2 child classes to AbfsLease as AbfsInfiniteLeaseV1 and
AbfsApiBundledLease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 605675)
Time Spent: 2h 50m (was: 2h 40m)
> ABFS: Introduce Lease Operations with Append to provide single writer
> semantics
> ---
>
> Key: HADOOP-17590
> URL: https://issues.apache.org/jira/browse/HADOOP-17590
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Sneha Varma
>Assignee: Sneha Varma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> The lease operations will be introduced as part of Append, Flush to ensure
> the single writer semantics.
>
> Details:
> Acquire Lease will be introduced in Create, Auto-Renew, Acquire will be added
> to Append & Release, Auto-Renew, Acquire in Flush.
>
> Duration the creation of the file the lease will be acquired, as part of
> appends the lease will be auto-renewed & the lease can be released as part of
> flush.
>
> By default the lease duration will be of 60 seconds.
> "fs.azure.write.enforcelease" & "fs.azure.write.lease.duration" two configs
> will be introduced.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #3026: HADOOP-17590 ABFS: Introduce Lease Operations with Append to provide single writer semantics
snvijaya commented on a change in pull request #3026:
URL: https://github.com/apache/hadoop/pull/3026#discussion_r643842160
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##
@@ -1698,14 +1703,29 @@ private void updateInfiniteLeaseDirs() {
this.azureInfiniteLeaseDirSet.remove("");
}
- private AbfsLease maybeCreateLease(String relativePath)
+ private AbfsLease maybeCreateFiniteLease(String relativePath, boolean
isNamespaceEnabled)
throws AzureBlobFileSystemException {
boolean enableInfiniteLease = isInfiniteLeaseKey(relativePath);
-if (!enableInfiniteLease) {
- return null;
+AbfsLease lease = null;
+if (!enableInfiniteLease && abfsConfiguration.isLeaseEnforced() &&
isNamespaceEnabled) {
+ lease = new AbfsLease(client, relativePath, false);
Review comment:
Are there scenarios where a finite lease created needs to explicitly
call on acquireLease API ? If that dependency isnt there, it would be better to
create 2 child classes to AbfsLease as AbfsInfiniteLeaseV1 and
AbfsApiBundledLease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #3062: HDFS-16048. RBF: Print network topology on the router web
goiri commented on a change in pull request #3062: URL: https://github.com/apache/hadoop/pull/3062#discussion_r644487404 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NetworkTopologyServlet.java ## @@ -171,7 +171,7 @@ private void printTextFormat(PrintStream stream, Map leaves, + private void printTopology(PrintStream stream, List leaves, Review comment: Why not just override? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17727) Modularize docker images
[ https://issues.apache.org/jira/browse/HADOOP-17727?focusedWorklogId=605656&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605656 ] ASF GitHub Bot logged work on HADOOP-17727: --- Author: ASF GitHub Bot Created on: 03/Jun/21 04:57 Start Date: 03/Jun/21 04:57 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #3043: URL: https://github.com/apache/hadoop/pull/3043#issuecomment-853562148 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3043/23/console in case of problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 605656) Time Spent: 8h 40m (was: 8.5h) > Modularize docker images > > > Key: HADOOP-17727 > URL: https://issues.apache.org/jira/browse/HADOOP-17727 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.4.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: pull-request-available > Time Spent: 8h 40m > Remaining Estimate: 0h > > We're now creating the *Dockerfile*s for different platforms. We need a way > to manage the packages in a clean way as maintaining the packages for all the > different environments becomes cumbersome. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3043: HADOOP-17727. Modularize docker images
hadoop-yetus commented on pull request #3043: URL: https://github.com/apache/hadoop/pull/3043#issuecomment-853562148 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3043/23/console in case of problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #2887: HDFS-15960 RBF: Router should talk to namenode with security context.
goiri commented on a change in pull request #2887:
URL: https://github.com/apache/hadoop/pull/2887#discussion_r644478595
##
File path:
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/NamenodeHeartbeatService.java
##
@@ -170,7 +172,20 @@ protected void serviceInit(Configuration configuration)
throws Exception {
@Override
public void periodicInvoke() {
-updateState();
+try {
+ SecurityUtil.doAsCurrentUser(
+ new PrivilegedExceptionAction() {
+@Override
+public Object run() {
+ updateState();
+ return null;
+}
+ });
+} catch (IOException e) {
+ // Generic error that we don't know about
+ LOG.error("Unexpected exception while communicating with {}: {}",
Review comment:
#3 sounds reasonable, do you mind giving it a try in this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17742) DistCp: distcp fail when copying to ftp filesystem
[
https://issues.apache.org/jira/browse/HADOOP-17742?focusedWorklogId=605653&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605653
]
ASF GitHub Bot logged work on HADOOP-17742:
---
Author: ASF GitHub Bot
Created on: 03/Jun/21 04:37
Start Date: 03/Jun/21 04:37
Worklog Time Spent: 10m
Work Description: zhaomin1423 commented on pull request #3067:
URL: https://github.com/apache/hadoop/pull/3067#issuecomment-853555466
This is problematic, I will resubmit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 605653)
Time Spent: 40m (was: 0.5h)
> DistCp: distcp fail when copying to ftp filesystem
> --
>
> Key: HADOOP-17742
> URL: https://issues.apache.org/jira/browse/HADOOP-17742
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 3.3.0
>Reporter: jerry
>Assignee: jerry
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When I copy file from a ftp filesystem to other ftp filesystem by distCp,
> there are two problems. Firstly, the tmp target path can't be found in the
> exception information. After my test, it was caused by the comma. Then when
> the uri of target path with schema, host, port can't be rename because of the
> ioexception.
> {code:java}
> // exception information
> java.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not existjava.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not exist at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
> {code}
> When I update the tempFileName from '.distcp.tmp.XXX' in
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#getTempFile to
> 'distcp.tmp.XXX', the question one is gone.
> The targetPath can be renamed when only the path of uri exists in
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#doCopy.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] zhaomin1423 commented on pull request #3067: HADOOP-17742. fix distcp fail when copying to ftp filesystem
zhaomin1423 commented on pull request #3067: URL: https://github.com/apache/hadoop/pull/3067#issuecomment-853555466 This is problematic, I will resubmit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17742) DistCp: distcp fail when copying to ftp filesystem
[
https://issues.apache.org/jira/browse/HADOOP-17742?focusedWorklogId=605652&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605652
]
ASF GitHub Bot logged work on HADOOP-17742:
---
Author: ASF GitHub Bot
Created on: 03/Jun/21 04:36
Start Date: 03/Jun/21 04:36
Worklog Time Spent: 10m
Work Description: zhaomin1423 closed pull request #3067:
URL: https://github.com/apache/hadoop/pull/3067
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 605652)
Time Spent: 0.5h (was: 20m)
> DistCp: distcp fail when copying to ftp filesystem
> --
>
> Key: HADOOP-17742
> URL: https://issues.apache.org/jira/browse/HADOOP-17742
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 3.3.0
>Reporter: jerry
>Assignee: jerry
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When I copy file from a ftp filesystem to other ftp filesystem by distCp,
> there are two problems. Firstly, the tmp target path can't be found in the
> exception information. After my test, it was caused by the comma. Then when
> the uri of target path with schema, host, port can't be rename because of the
> ioexception.
> {code:java}
> // exception information
> java.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not existjava.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not exist at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
> {code}
> When I update the tempFileName from '.distcp.tmp.XXX' in
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#getTempFile to
> 'distcp.tmp.XXX', the question one is gone.
> The targetPath can be renamed when only the path of uri exists in
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#doCopy.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] zhaomin1423 closed pull request #3067: HADOOP-17742. fix distcp fail when copying to ftp filesystem
zhaomin1423 closed pull request #3067: URL: https://github.com/apache/hadoop/pull/3067 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] oojas commented on pull request #3059: #HDFS-13729 Removed extra space
oojas commented on pull request #3059: URL: https://github.com/apache/hadoop/pull/3059#issuecomment-853504454 > That space is in the comment, right? What problem it is creating? NO its in the readme documentation -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17028) ViewFS should initialize target filesystems lazily
[ https://issues.apache.org/jira/browse/HADOOP-17028?focusedWorklogId=605626&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605626 ] ASF GitHub Bot logged work on HADOOP-17028: --- Author: ASF GitHub Bot Created on: 03/Jun/21 01:48 Start Date: 03/Jun/21 01:48 Worklog Time Spent: 10m Work Description: abhishekdas99 commented on a change in pull request #2260: URL: https://github.com/apache/hadoop/pull/2260#discussion_r644426716 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java ## @@ -37,6 +37,7 @@ import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.fs.UnsupportedFileSystemException; +import org.apache.hadoop.fs.impl.FunctionsRaisingIOE.FunctionRaisingIOE; Review comment: replaced `org.apache.hadoop.fs.impl.FunctionsRaisingIOE.FunctionRaisingIOE` with `org.apache.hadoop.util.functional.FunctionRaisingIOE` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 605626) Time Spent: 3h 50m (was: 3h 40m) > ViewFS should initialize target filesystems lazily > -- > > Key: HADOOP-17028 > URL: https://issues.apache.org/jira/browse/HADOOP-17028 > Project: Hadoop Common > Issue Type: Bug > Components: client-mounts, fs, viewfs >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Assignee: Abhishek Das >Priority: Major > Labels: pull-request-available > Time Spent: 3h 50m > Remaining Estimate: 0h > > Currently viewFS initialize all configured target filesystems when > viewfs#init itself. > Some target file system initialization involve creating heavy objects and > proxy connections. Ex: DistributedFileSystem#initialize will create DFSClient > object which will create proxy connections to NN etc. > For example: if ViewFS configured with 10 target fs with hdfs uri and 2 > targets with s3a. > If one of the client only work with s3a target, But ViewFS will initialize > all targets irrespective of what clients interested to work with. That means, > here client will create 10 DFS initializations and 2 s3a initializations. Its > unnecessary to have DFS initialization here. So, it will be a good idea to > initialize the target fs only when first time usage call come to particular > target fs scheme. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] abhishekdas99 commented on a change in pull request #2260: HADOOP-17028. ViewFS should initialize mounted target filesystems lazily
abhishekdas99 commented on a change in pull request #2260: URL: https://github.com/apache/hadoop/pull/2260#discussion_r644426716 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java ## @@ -37,6 +37,7 @@ import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.fs.UnsupportedFileSystemException; +import org.apache.hadoop.fs.impl.FunctionsRaisingIOE.FunctionRaisingIOE; Review comment: replaced `org.apache.hadoop.fs.impl.FunctionsRaisingIOE.FunctionRaisingIOE` with `org.apache.hadoop.util.functional.FunctionRaisingIOE` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17028) ViewFS should initialize target filesystems lazily
[ https://issues.apache.org/jira/browse/HADOOP-17028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356063#comment-17356063 ] Konstantin Shvachko commented on HADOOP-17028: -- Left minor comments on the PR. The approach looks reasonable to me. The important thing is to get a Jenkins build. Looks like it could not run. May be you fork got stale. You should probably rebase on current trunk. > ViewFS should initialize target filesystems lazily > -- > > Key: HADOOP-17028 > URL: https://issues.apache.org/jira/browse/HADOOP-17028 > Project: Hadoop Common > Issue Type: Bug > Components: client-mounts, fs, viewfs >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Assignee: Abhishek Das >Priority: Major > Labels: pull-request-available > Time Spent: 3h 40m > Remaining Estimate: 0h > > Currently viewFS initialize all configured target filesystems when > viewfs#init itself. > Some target file system initialization involve creating heavy objects and > proxy connections. Ex: DistributedFileSystem#initialize will create DFSClient > object which will create proxy connections to NN etc. > For example: if ViewFS configured with 10 target fs with hdfs uri and 2 > targets with s3a. > If one of the client only work with s3a target, But ViewFS will initialize > all targets irrespective of what clients interested to work with. That means, > here client will create 10 DFS initializations and 2 s3a initializations. Its > unnecessary to have DFS initialization here. So, it will be a good idea to > initialize the target fs only when first time usage call come to particular > target fs scheme. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17028) ViewFS should initialize target filesystems lazily
[
https://issues.apache.org/jira/browse/HADOOP-17028?focusedWorklogId=605623&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605623
]
ASF GitHub Bot logged work on HADOOP-17028:
---
Author: ASF GitHub Bot
Created on: 03/Jun/21 00:44
Start Date: 03/Jun/21 00:44
Worklog Time Spent: 10m
Work Description: shvachko commented on a change in pull request #2260:
URL: https://github.com/apache/hadoop/pull/2260#discussion_r644407043
##
File path:
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java
##
@@ -1428,4 +1430,45 @@ public void testGetContentSummaryWithFileInLocalFS()
throws Exception {
summaryAfter.getLength());
}
}
+
+ @Test
+ public void testTargetFileSystemLazyInitialization() throws Exception {
+final String clusterName = "cluster" + new Random().nextInt();
+Configuration config = new Configuration(conf);
+config.setBoolean(CONFIG_VIEWFS_ENABLE_INNER_CACHE, false);
+config.setClass("fs.mockfs.impl",
+TestChRootedFileSystem.MockFileSystem.class, FileSystem.class);
+ConfigUtil.addLink(config, clusterName, "/user",
+URI.create("mockfs://mockauth1/mockpath"));
+ConfigUtil.addLink(config, clusterName,
+"/mock", URI.create("mockfs://mockauth/mockpath"));
+
+final int cacheSize = TestFileUtil.getCacheSize();
Review comment:
You can call `closeAll()` here, which clears the cache completely. And
the you can compare with 0 in the assert.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 605623)
Time Spent: 3h 40m (was: 3.5h)
> ViewFS should initialize target filesystems lazily
> --
>
> Key: HADOOP-17028
> URL: https://issues.apache.org/jira/browse/HADOOP-17028
> Project: Hadoop Common
> Issue Type: Bug
> Components: client-mounts, fs, viewfs
>Affects Versions: 3.2.1
>Reporter: Uma Maheswara Rao G
>Assignee: Abhishek Das
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> Currently viewFS initialize all configured target filesystems when
> viewfs#init itself.
> Some target file system initialization involve creating heavy objects and
> proxy connections. Ex: DistributedFileSystem#initialize will create DFSClient
> object which will create proxy connections to NN etc.
> For example: if ViewFS configured with 10 target fs with hdfs uri and 2
> targets with s3a.
> If one of the client only work with s3a target, But ViewFS will initialize
> all targets irrespective of what clients interested to work with. That means,
> here client will create 10 DFS initializations and 2 s3a initializations. Its
> unnecessary to have DFS initialization here. So, it will be a good idea to
> initialize the target fs only when first time usage call come to particular
> target fs scheme.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] shvachko commented on a change in pull request #2260: HADOOP-17028. ViewFS should initialize mounted target filesystems lazily
shvachko commented on a change in pull request #2260:
URL: https://github.com/apache/hadoop/pull/2260#discussion_r644407043
##
File path:
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java
##
@@ -1428,4 +1430,45 @@ public void testGetContentSummaryWithFileInLocalFS()
throws Exception {
summaryAfter.getLength());
}
}
+
+ @Test
+ public void testTargetFileSystemLazyInitialization() throws Exception {
+final String clusterName = "cluster" + new Random().nextInt();
+Configuration config = new Configuration(conf);
+config.setBoolean(CONFIG_VIEWFS_ENABLE_INNER_CACHE, false);
+config.setClass("fs.mockfs.impl",
+TestChRootedFileSystem.MockFileSystem.class, FileSystem.class);
+ConfigUtil.addLink(config, clusterName, "/user",
+URI.create("mockfs://mockauth1/mockpath"));
+ConfigUtil.addLink(config, clusterName,
+"/mock", URI.create("mockfs://mockauth/mockpath"));
+
+final int cacheSize = TestFileUtil.getCacheSize();
Review comment:
You can call `closeAll()` here, which clears the cache completely. And
the you can compare with 0 in the assert.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17742) DistCp: distcp fail when copying to ftp filesystem
[
https://issues.apache.org/jira/browse/HADOOP-17742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jerry updated HADOOP-17742:
---
Description:
When I copy file from a ftp filesystem to other ftp filesystem by distCp, there
are two problems. Firstly, the tmp target path can't be found in the exception
information. After my test, it was caused by the comma. Then when the uri of
target path with schema, host, port can't be rename because of the ioexception.
{code:java}
// exception information
java.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not existjava.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not exist at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
{code}
When I update the tempFileName from '.distcp.tmp.XXX' in
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#getTempFile to
'distcp.tmp.XXX', the question one is gone.
The targetPath can be renamed when only the path of uri exists in
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#doCopy.
was:
When I copy file from a ftp filesystem to other ftp filesystem by distCp, there
are two problems. Firstly, the tmp target path can't be found in the exception
information. After my test, it was caused by the comma. Then when the uri of
target path with schema, host, port can't be rename because of the ioexception.
{code:java}
// exception information
java.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not existjava.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not exist at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
{code}
When I update the tempFileName from '.distcp.tmp.XXX' in
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#getTempFile to
'.distcp.tmp.XXX', the question one is gone.
The targetPath can be renamed when only the path of uri exists in
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#doCopy.
> DistCp: distcp fail when copying to ftp filesystem
> --
>
> Key: HADOOP-17742
> URL: https://issues.apache.org/jira/browse/HADOOP-17742
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 3.3.0
>Reporter: jerry
>Assignee: jerry
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When I copy file from a ftp filesystem to other ftp filesystem by distCp,
> there are two problems. Firstly, the tmp target path can't be found in the
> exception information. After my test, it was caused by the comma. Then when
> the uri of target path with schema, host, port can't be rename because of the
> ioexception.
> {code:java}
> // exception information
> java.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not existjava.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not exist at
> org.apa
[jira] [Work logged] (HADOOP-17028) ViewFS should initialize target filesystems lazily
[ https://issues.apache.org/jira/browse/HADOOP-17028?focusedWorklogId=605619&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605619 ] ASF GitHub Bot logged work on HADOOP-17028: --- Author: ASF GitHub Bot Created on: 03/Jun/21 00:02 Start Date: 03/Jun/21 00:02 Worklog Time Spent: 10m Work Description: shvachko commented on a change in pull request #2260: URL: https://github.com/apache/hadoop/pull/2260#discussion_r644393901 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java ## @@ -37,6 +37,7 @@ import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.fs.UnsupportedFileSystemException; +import org.apache.hadoop.fs.impl.FunctionsRaisingIOE.FunctionRaisingIOE; Review comment: It looks like @steveloughran deprecated this class later after his comment. Don't know what prompted the refactoring, but `import org.apache.hadoop.fs.impl.FunctionsRaisingIOE.FunctionRaisingIOE` should be now `org.apache.hadoop.util.functional.FunctionRaisingIOE` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 605619) Time Spent: 3.5h (was: 3h 20m) > ViewFS should initialize target filesystems lazily > -- > > Key: HADOOP-17028 > URL: https://issues.apache.org/jira/browse/HADOOP-17028 > Project: Hadoop Common > Issue Type: Bug > Components: client-mounts, fs, viewfs >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Assignee: Abhishek Das >Priority: Major > Labels: pull-request-available > Time Spent: 3.5h > Remaining Estimate: 0h > > Currently viewFS initialize all configured target filesystems when > viewfs#init itself. > Some target file system initialization involve creating heavy objects and > proxy connections. Ex: DistributedFileSystem#initialize will create DFSClient > object which will create proxy connections to NN etc. > For example: if ViewFS configured with 10 target fs with hdfs uri and 2 > targets with s3a. > If one of the client only work with s3a target, But ViewFS will initialize > all targets irrespective of what clients interested to work with. That means, > here client will create 10 DFS initializations and 2 s3a initializations. Its > unnecessary to have DFS initialization here. So, it will be a good idea to > initialize the target fs only when first time usage call come to particular > target fs scheme. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] shvachko commented on a change in pull request #2260: HADOOP-17028. ViewFS should initialize mounted target filesystems lazily
shvachko commented on a change in pull request #2260: URL: https://github.com/apache/hadoop/pull/2260#discussion_r644393901 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java ## @@ -37,6 +37,7 @@ import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.fs.UnsupportedFileSystemException; +import org.apache.hadoop.fs.impl.FunctionsRaisingIOE.FunctionRaisingIOE; Review comment: It looks like @steveloughran deprecated this class later after his comment. Don't know what prompted the refactoring, but `import org.apache.hadoop.fs.impl.FunctionsRaisingIOE.FunctionRaisingIOE` should be now `org.apache.hadoop.util.functional.FunctionRaisingIOE` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17742) DistCp: distcp fail when copying to ftp filesystem
[
https://issues.apache.org/jira/browse/HADOOP-17742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356049#comment-17356049
]
jerry commented on HADOOP-17742:
Thanks, "-direct" option can avoid, but the writing process is not atomic.
> DistCp: distcp fail when copying to ftp filesystem
> --
>
> Key: HADOOP-17742
> URL: https://issues.apache.org/jira/browse/HADOOP-17742
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 3.3.0
>Reporter: jerry
>Assignee: jerry
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When I copy file from a ftp filesystem to other ftp filesystem by distCp,
> there are two problems. Firstly, the tmp target path can't be found in the
> exception information. After my test, it was caused by the comma. Then when
> the uri of target path with schema, host, port can't be rename because of the
> ioexception.
> {code:java}
> // exception information
> java.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not existjava.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not exist at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
> {code}
> When I update the tempFileName from '.distcp.tmp.XXX' in
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#getTempFile to
> '.distcp.tmp.XXX', the question one is gone.
> The targetPath can be renamed when only the path of uri exists in
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#doCopy.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17742) DistCp: distcp fail when copying to ftp filesystem
[
https://issues.apache.org/jira/browse/HADOOP-17742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356048#comment-17356048
]
jerry commented on HADOOP-17742:
the ftp was deployed by docker, the image is fauria/vsftpd, the link is
https://hub.docker.com/r/fauria/vsftpd, This Docker container implements a
vsftpd server, with the following features:
Centos 7 base image.
vsftpd 3.0
Virtual users
Passive mode
Logging to a file or STDOUT.
> DistCp: distcp fail when copying to ftp filesystem
> --
>
> Key: HADOOP-17742
> URL: https://issues.apache.org/jira/browse/HADOOP-17742
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 3.3.0
>Reporter: jerry
>Assignee: jerry
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When I copy file from a ftp filesystem to other ftp filesystem by distCp,
> there are two problems. Firstly, the tmp target path can't be found in the
> exception information. After my test, it was caused by the comma. Then when
> the uri of target path with schema, host, port can't be rename because of the
> ioexception.
> {code:java}
> // exception information
> java.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not existjava.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not exist at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
> {code}
> When I update the tempFileName from '.distcp.tmp.XXX' in
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#getTempFile to
> '.distcp.tmp.XXX', the question one is gone.
> The targetPath can be renamed when only the path of uri exists in
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#doCopy.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17028) ViewFS should initialize target filesystems lazily
[ https://issues.apache.org/jira/browse/HADOOP-17028?focusedWorklogId=605547&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605547 ] ASF GitHub Bot logged work on HADOOP-17028: --- Author: ASF GitHub Bot Created on: 02/Jun/21 20:54 Start Date: 02/Jun/21 20:54 Worklog Time Spent: 10m Work Description: abhishekdas99 edited a comment on pull request #2260: URL: https://github.com/apache/hadoop/pull/2260#issuecomment-853374990 @umamaheswararao @steveloughran bumping up the conversation on this thread. Recently we have faced lot of issues (OOM, fs initialization time going up) because of not having lazy initialization. We see these issues where we have 10k mount points and ViewFileSystem initializes all the mount points during initialization. Want to get a review on this one. cc @virajith @shvachko -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 605547) Time Spent: 3h 20m (was: 3h 10m) > ViewFS should initialize target filesystems lazily > -- > > Key: HADOOP-17028 > URL: https://issues.apache.org/jira/browse/HADOOP-17028 > Project: Hadoop Common > Issue Type: Bug > Components: client-mounts, fs, viewfs >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Assignee: Abhishek Das >Priority: Major > Labels: pull-request-available > Time Spent: 3h 20m > Remaining Estimate: 0h > > Currently viewFS initialize all configured target filesystems when > viewfs#init itself. > Some target file system initialization involve creating heavy objects and > proxy connections. Ex: DistributedFileSystem#initialize will create DFSClient > object which will create proxy connections to NN etc. > For example: if ViewFS configured with 10 target fs with hdfs uri and 2 > targets with s3a. > If one of the client only work with s3a target, But ViewFS will initialize > all targets irrespective of what clients interested to work with. That means, > here client will create 10 DFS initializations and 2 s3a initializations. Its > unnecessary to have DFS initialization here. So, it will be a good idea to > initialize the target fs only when first time usage call come to particular > target fs scheme. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] abhishekdas99 edited a comment on pull request #2260: HADOOP-17028. ViewFS should initialize mounted target filesystems lazily
abhishekdas99 edited a comment on pull request #2260: URL: https://github.com/apache/hadoop/pull/2260#issuecomment-853374990 @umamaheswararao @steveloughran bumping up the conversation on this thread. Recently we have faced lot of issues (OOM, fs initialization time going up) because of not having lazy initialization. We see these issues where we have 10k mount points and ViewFileSystem initializes all the mount points during initialization. Want to get a review on this one. cc @virajith @shvachko -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17028) ViewFS should initialize target filesystems lazily
[ https://issues.apache.org/jira/browse/HADOOP-17028?focusedWorklogId=605546&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605546 ] ASF GitHub Bot logged work on HADOOP-17028: --- Author: ASF GitHub Bot Created on: 02/Jun/21 20:52 Start Date: 02/Jun/21 20:52 Worklog Time Spent: 10m Work Description: abhishekdas99 commented on pull request #2260: URL: https://github.com/apache/hadoop/pull/2260#issuecomment-853374990 @umamaheswararao @steveloughran bumping up the conversation on this thread. Recently we have faced lot of issues (OOM, fs initialization time going up) because of not having lazy initialization. Want to get a review on this one. cc @virajith @shvachko -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 605546) Time Spent: 3h 10m (was: 3h) > ViewFS should initialize target filesystems lazily > -- > > Key: HADOOP-17028 > URL: https://issues.apache.org/jira/browse/HADOOP-17028 > Project: Hadoop Common > Issue Type: Bug > Components: client-mounts, fs, viewfs >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Assignee: Abhishek Das >Priority: Major > Labels: pull-request-available > Time Spent: 3h 10m > Remaining Estimate: 0h > > Currently viewFS initialize all configured target filesystems when > viewfs#init itself. > Some target file system initialization involve creating heavy objects and > proxy connections. Ex: DistributedFileSystem#initialize will create DFSClient > object which will create proxy connections to NN etc. > For example: if ViewFS configured with 10 target fs with hdfs uri and 2 > targets with s3a. > If one of the client only work with s3a target, But ViewFS will initialize > all targets irrespective of what clients interested to work with. That means, > here client will create 10 DFS initializations and 2 s3a initializations. Its > unnecessary to have DFS initialization here. So, it will be a good idea to > initialize the target fs only when first time usage call come to particular > target fs scheme. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] abhishekdas99 commented on pull request #2260: HADOOP-17028. ViewFS should initialize mounted target filesystems lazily
abhishekdas99 commented on pull request #2260: URL: https://github.com/apache/hadoop/pull/2260#issuecomment-853374990 @umamaheswararao @steveloughran bumping up the conversation on this thread. Recently we have faced lot of issues (OOM, fs initialization time going up) because of not having lazy initialization. Want to get a review on this one. cc @virajith @shvachko -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17028) ViewFS should initialize target filesystems lazily
[
https://issues.apache.org/jira/browse/HADOOP-17028?focusedWorklogId=605545&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605545
]
ASF GitHub Bot logged work on HADOOP-17028:
---
Author: ASF GitHub Bot
Created on: 02/Jun/21 20:50
Start Date: 02/Jun/21 20:50
Worklog Time Spent: 10m
Work Description: abhishekdas99 commented on a change in pull request
#2260:
URL: https://github.com/apache/hadoop/pull/2260#discussion_r644310790
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
##
@@ -936,8 +956,13 @@ public void setWriteChecksum(final boolean writeChecksum) {
fsState.getMountPoints();
Set children = new HashSet();
for (InodeTree.MountPoint mountPoint : mountPoints) {
- FileSystem targetFs = mountPoint.target.targetFileSystem;
- children.addAll(Arrays.asList(targetFs.getChildFileSystems()));
+ try {
+FileSystem targetFs = mountPoint.target.getTargetFileSystem();
+children.addAll(Arrays.asList(targetFs.getChildFileSystems()));
+ } catch (IOException ex) {
+LOG.error("Could not add child filesystems "
++ "for source path " + mountPoint.src);
Review comment:
Changed the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 605545)
Time Spent: 3h (was: 2h 50m)
> ViewFS should initialize target filesystems lazily
> --
>
> Key: HADOOP-17028
> URL: https://issues.apache.org/jira/browse/HADOOP-17028
> Project: Hadoop Common
> Issue Type: Bug
> Components: client-mounts, fs, viewfs
>Affects Versions: 3.2.1
>Reporter: Uma Maheswara Rao G
>Assignee: Abhishek Das
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Currently viewFS initialize all configured target filesystems when
> viewfs#init itself.
> Some target file system initialization involve creating heavy objects and
> proxy connections. Ex: DistributedFileSystem#initialize will create DFSClient
> object which will create proxy connections to NN etc.
> For example: if ViewFS configured with 10 target fs with hdfs uri and 2
> targets with s3a.
> If one of the client only work with s3a target, But ViewFS will initialize
> all targets irrespective of what clients interested to work with. That means,
> here client will create 10 DFS initializations and 2 s3a initializations. Its
> unnecessary to have DFS initialization here. So, it will be a good idea to
> initialize the target fs only when first time usage call come to particular
> target fs scheme.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] abhishekdas99 commented on a change in pull request #2260: HADOOP-17028. ViewFS should initialize mounted target filesystems lazily
abhishekdas99 commented on a change in pull request #2260:
URL: https://github.com/apache/hadoop/pull/2260#discussion_r644310790
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
##
@@ -936,8 +956,13 @@ public void setWriteChecksum(final boolean writeChecksum) {
fsState.getMountPoints();
Set children = new HashSet();
for (InodeTree.MountPoint mountPoint : mountPoints) {
- FileSystem targetFs = mountPoint.target.targetFileSystem;
- children.addAll(Arrays.asList(targetFs.getChildFileSystems()));
+ try {
+FileSystem targetFs = mountPoint.target.getTargetFileSystem();
+children.addAll(Arrays.asList(targetFs.getChildFileSystems()));
+ } catch (IOException ex) {
+LOG.error("Could not add child filesystems "
++ "for source path " + mountPoint.src);
Review comment:
Changed the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17028) ViewFS should initialize target filesystems lazily
[
https://issues.apache.org/jira/browse/HADOOP-17028?focusedWorklogId=605542&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605542
]
ASF GitHub Bot logged work on HADOOP-17028:
---
Author: ASF GitHub Bot
Created on: 02/Jun/21 20:49
Start Date: 02/Jun/21 20:49
Worklog Time Spent: 10m
Work Description: abhishekdas99 commented on a change in pull request
#2260:
URL: https://github.com/apache/hadoop/pull/2260#discussion_r644310415
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
##
@@ -284,7 +288,20 @@ boolean isInternalDir() {
return false;
}
-public T getTargetFileSystem() {
+/**
+ * Gets lazily loaded instance of FileSystem
+ * @return An Initialized instance of T
+ * @throws IOException
+ */
+public T getTargetFileSystem() throws IOException {
+ if (targetFileSystem != null)
+return targetFileSystem;
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 605542)
Time Spent: 2.5h (was: 2h 20m)
> ViewFS should initialize target filesystems lazily
> --
>
> Key: HADOOP-17028
> URL: https://issues.apache.org/jira/browse/HADOOP-17028
> Project: Hadoop Common
> Issue Type: Bug
> Components: client-mounts, fs, viewfs
>Affects Versions: 3.2.1
>Reporter: Uma Maheswara Rao G
>Assignee: Abhishek Das
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Currently viewFS initialize all configured target filesystems when
> viewfs#init itself.
> Some target file system initialization involve creating heavy objects and
> proxy connections. Ex: DistributedFileSystem#initialize will create DFSClient
> object which will create proxy connections to NN etc.
> For example: if ViewFS configured with 10 target fs with hdfs uri and 2
> targets with s3a.
> If one of the client only work with s3a target, But ViewFS will initialize
> all targets irrespective of what clients interested to work with. That means,
> here client will create 10 DFS initializations and 2 s3a initializations. Its
> unnecessary to have DFS initialization here. So, it will be a good idea to
> initialize the target fs only when first time usage call come to particular
> target fs scheme.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17028) ViewFS should initialize target filesystems lazily
[
https://issues.apache.org/jira/browse/HADOOP-17028?focusedWorklogId=605544&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605544
]
ASF GitHub Bot logged work on HADOOP-17028:
---
Author: ASF GitHub Bot
Created on: 02/Jun/21 20:49
Start Date: 02/Jun/21 20:49
Worklog Time Spent: 10m
Work Description: abhishekdas99 commented on a change in pull request
#2260:
URL: https://github.com/apache/hadoop/pull/2260#discussion_r644310654
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
##
@@ -855,7 +861,11 @@ public void setVerifyChecksum(final boolean
verifyChecksum) {
List> mountPoints =
fsState.getMountPoints();
for (InodeTree.MountPoint mount : mountPoints) {
- mount.target.targetFileSystem.setVerifyChecksum(verifyChecksum);
+ try {
+mount.target.getTargetFileSystem().setVerifyChecksum(verifyChecksum);
+ } catch (IOException ex) {
+LOG.error("Could not set verifyChecksum for source path " + mount.src);
+ }
Review comment:
Changed the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 605544)
Time Spent: 2h 50m (was: 2h 40m)
> ViewFS should initialize target filesystems lazily
> --
>
> Key: HADOOP-17028
> URL: https://issues.apache.org/jira/browse/HADOOP-17028
> Project: Hadoop Common
> Issue Type: Bug
> Components: client-mounts, fs, viewfs
>Affects Versions: 3.2.1
>Reporter: Uma Maheswara Rao G
>Assignee: Abhishek Das
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Currently viewFS initialize all configured target filesystems when
> viewfs#init itself.
> Some target file system initialization involve creating heavy objects and
> proxy connections. Ex: DistributedFileSystem#initialize will create DFSClient
> object which will create proxy connections to NN etc.
> For example: if ViewFS configured with 10 target fs with hdfs uri and 2
> targets with s3a.
> If one of the client only work with s3a target, But ViewFS will initialize
> all targets irrespective of what clients interested to work with. That means,
> here client will create 10 DFS initializations and 2 s3a initializations. Its
> unnecessary to have DFS initialization here. So, it will be a good idea to
> initialize the target fs only when first time usage call come to particular
> target fs scheme.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17028) ViewFS should initialize target filesystems lazily
[
https://issues.apache.org/jira/browse/HADOOP-17028?focusedWorklogId=605541&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605541
]
ASF GitHub Bot logged work on HADOOP-17028:
---
Author: ASF GitHub Bot
Created on: 02/Jun/21 20:49
Start Date: 02/Jun/21 20:49
Worklog Time Spent: 10m
Work Description: abhishekdas99 commented on a change in pull request
#2260:
URL: https://github.com/apache/hadoop/pull/2260#discussion_r644310313
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
##
@@ -284,7 +288,20 @@ boolean isInternalDir() {
return false;
}
-public T getTargetFileSystem() {
+/**
+ * Gets lazily loaded instance of FileSystem
Review comment:
added . at the end of the line
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 605541)
Time Spent: 2h 20m (was: 2h 10m)
> ViewFS should initialize target filesystems lazily
> --
>
> Key: HADOOP-17028
> URL: https://issues.apache.org/jira/browse/HADOOP-17028
> Project: Hadoop Common
> Issue Type: Bug
> Components: client-mounts, fs, viewfs
>Affects Versions: 3.2.1
>Reporter: Uma Maheswara Rao G
>Assignee: Abhishek Das
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Currently viewFS initialize all configured target filesystems when
> viewfs#init itself.
> Some target file system initialization involve creating heavy objects and
> proxy connections. Ex: DistributedFileSystem#initialize will create DFSClient
> object which will create proxy connections to NN etc.
> For example: if ViewFS configured with 10 target fs with hdfs uri and 2
> targets with s3a.
> If one of the client only work with s3a target, But ViewFS will initialize
> all targets irrespective of what clients interested to work with. That means,
> here client will create 10 DFS initializations and 2 s3a initializations. Its
> unnecessary to have DFS initialization here. So, it will be a good idea to
> initialize the target fs only when first time usage call come to particular
> target fs scheme.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17028) ViewFS should initialize target filesystems lazily
[
https://issues.apache.org/jira/browse/HADOOP-17028?focusedWorklogId=605543&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605543
]
ASF GitHub Bot logged work on HADOOP-17028:
---
Author: ASF GitHub Bot
Created on: 02/Jun/21 20:49
Start Date: 02/Jun/21 20:49
Worklog Time Spent: 10m
Work Description: abhishekdas99 commented on a change in pull request
#2260:
URL: https://github.com/apache/hadoop/pull/2260#discussion_r644310547
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
##
@@ -284,7 +288,20 @@ boolean isInternalDir() {
return false;
}
-public T getTargetFileSystem() {
+/**
+ * Gets lazily loaded instance of FileSystem
+ * @return An Initialized instance of T
+ * @throws IOException
+ */
+public T getTargetFileSystem() throws IOException {
+ if (targetFileSystem != null)
+return targetFileSystem;
+
+ if (targetDirLinkList.length == 1) {
+synchronized (this) {
+ targetFileSystem = fileSystemInitFunc.apply(targetDirLinkList[0]);
Review comment:
Added extra check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 605543)
Time Spent: 2h 40m (was: 2.5h)
> ViewFS should initialize target filesystems lazily
> --
>
> Key: HADOOP-17028
> URL: https://issues.apache.org/jira/browse/HADOOP-17028
> Project: Hadoop Common
> Issue Type: Bug
> Components: client-mounts, fs, viewfs
>Affects Versions: 3.2.1
>Reporter: Uma Maheswara Rao G
>Assignee: Abhishek Das
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Currently viewFS initialize all configured target filesystems when
> viewfs#init itself.
> Some target file system initialization involve creating heavy objects and
> proxy connections. Ex: DistributedFileSystem#initialize will create DFSClient
> object which will create proxy connections to NN etc.
> For example: if ViewFS configured with 10 target fs with hdfs uri and 2
> targets with s3a.
> If one of the client only work with s3a target, But ViewFS will initialize
> all targets irrespective of what clients interested to work with. That means,
> here client will create 10 DFS initializations and 2 s3a initializations. Its
> unnecessary to have DFS initialization here. So, it will be a good idea to
> initialize the target fs only when first time usage call come to particular
> target fs scheme.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] abhishekdas99 commented on a change in pull request #2260: HADOOP-17028. ViewFS should initialize mounted target filesystems lazily
abhishekdas99 commented on a change in pull request #2260:
URL: https://github.com/apache/hadoop/pull/2260#discussion_r644310654
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
##
@@ -855,7 +861,11 @@ public void setVerifyChecksum(final boolean
verifyChecksum) {
List> mountPoints =
fsState.getMountPoints();
for (InodeTree.MountPoint mount : mountPoints) {
- mount.target.targetFileSystem.setVerifyChecksum(verifyChecksum);
+ try {
+mount.target.getTargetFileSystem().setVerifyChecksum(verifyChecksum);
+ } catch (IOException ex) {
+LOG.error("Could not set verifyChecksum for source path " + mount.src);
+ }
Review comment:
Changed the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] abhishekdas99 commented on a change in pull request #2260: HADOOP-17028. ViewFS should initialize mounted target filesystems lazily
abhishekdas99 commented on a change in pull request #2260:
URL: https://github.com/apache/hadoop/pull/2260#discussion_r644310547
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
##
@@ -284,7 +288,20 @@ boolean isInternalDir() {
return false;
}
-public T getTargetFileSystem() {
+/**
+ * Gets lazily loaded instance of FileSystem
+ * @return An Initialized instance of T
+ * @throws IOException
+ */
+public T getTargetFileSystem() throws IOException {
+ if (targetFileSystem != null)
+return targetFileSystem;
+
+ if (targetDirLinkList.length == 1) {
+synchronized (this) {
+ targetFileSystem = fileSystemInitFunc.apply(targetDirLinkList[0]);
Review comment:
Added extra check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] abhishekdas99 commented on a change in pull request #2260: HADOOP-17028. ViewFS should initialize mounted target filesystems lazily
abhishekdas99 commented on a change in pull request #2260:
URL: https://github.com/apache/hadoop/pull/2260#discussion_r644310415
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
##
@@ -284,7 +288,20 @@ boolean isInternalDir() {
return false;
}
-public T getTargetFileSystem() {
+/**
+ * Gets lazily loaded instance of FileSystem
+ * @return An Initialized instance of T
+ * @throws IOException
+ */
+public T getTargetFileSystem() throws IOException {
+ if (targetFileSystem != null)
+return targetFileSystem;
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] abhishekdas99 commented on a change in pull request #2260: HADOOP-17028. ViewFS should initialize mounted target filesystems lazily
abhishekdas99 commented on a change in pull request #2260:
URL: https://github.com/apache/hadoop/pull/2260#discussion_r644310313
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java
##
@@ -284,7 +288,20 @@ boolean isInternalDir() {
return false;
}
-public T getTargetFileSystem() {
+/**
+ * Gets lazily loaded instance of FileSystem
Review comment:
added . at the end of the line
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bolerio commented on a change in pull request #2887: HDFS-15960 RBF: Router should talk to namenode with security context.
bolerio commented on a change in pull request #2887:
URL: https://github.com/apache/hadoop/pull/2887#discussion_r644283148
##
File path:
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/NamenodeHeartbeatService.java
##
@@ -170,7 +172,20 @@ protected void serviceInit(Configuration configuration)
throws Exception {
@Override
public void periodicInvoke() {
-updateState();
+try {
+ SecurityUtil.doAsCurrentUser(
+ new PrivilegedExceptionAction() {
+@Override
+public Object run() {
+ updateState();
+ return null;
+}
+ });
+} catch (IOException e) {
+ // Generic error that we don't know about
+ LOG.error("Unexpected exception while communicating with {}: {}",
Review comment:
Hi @goiri , following up this. I was able to create a unit test that
reproduces the problem and demonstrate that the patch fixes it. However, there
is a challenge.
The failure is when the router calls the JMX endpoint which returns some
info stats in addition to the basic alive status which is obtained in a
separate RPC call. The failure is soft - logs the exception and continues,
without the information it tried to obtain. However that information is needed
later during load balancing, which is how the original bug was discovered.
Now, because the main interface capturing knowledge about a NN on the router
side (FederationNamenodeContext) does not contain these stats, there is no way
to write a unit test against it. There are some unit tests in that area that
mock this interface and I modified the mock to include stats, but then I have
to downcast to the mock object in the test which is very ugly.
So the options are: (1) accept this ugly downcast (2) don't write the test
and eventually if Hadoop has an integration test suite, cover the use case
there and (3) modify the FederationNamenodeContext to include the stats (see
MembershipState and MembershipStats class). My vote would be for (3) as those
stats seem essential to the operation of a federated cluster. It would be ok
not to make all of the numbers part of the public interface, but the fact that
we need stats about resource utilization should be part of the interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on pull request #3059: #HDFS-13729 Removed extra space
ayushtkn commented on pull request #3059: URL: https://github.com/apache/hadoop/pull/3059#issuecomment-853340317 That space is in the comment right? What problem it is creating? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xiaoyuyao commented on a change in pull request #3065: HDFS-13671. Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet
xiaoyuyao commented on a change in pull request #3065: URL: https://github.com/apache/hadoop/pull/3065#discussion_r644152306 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java ## @@ -188,8 +188,7 @@ public HeartbeatResponse sendHeartbeat(DatanodeRegistration registration, @Override public DatanodeCommand blockReport(DatanodeRegistration registration, - String poolId, StorageBlockReport[] reports, - BlockReportContext context) + String poolId, StorageBlockReport[] reports, BlockReportContext context) Review comment: NIT: unnecessary formatting change can be avoided. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17742) DistCp: distcp fail when copying to ftp filesystem
[
https://issues.apache.org/jira/browse/HADOOP-17742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17355855#comment-17355855
]
Steve Loughran commented on HADOOP-17742:
-
what was the destination FTP OS/version? I'd be surprised that a . prefix never
worked before.
The "-direct" option will avoid this problem entirely, BTW
> DistCp: distcp fail when copying to ftp filesystem
> --
>
> Key: HADOOP-17742
> URL: https://issues.apache.org/jira/browse/HADOOP-17742
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 3.3.0
>Reporter: jerry
>Assignee: jerry
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When I copy file from a ftp filesystem to other ftp filesystem by distCp,
> there are two problems. Firstly, the tmp target path can't be found in the
> exception information. After my test, it was caused by the comma. Then when
> the uri of target path with schema, host, port can't be rename because of the
> ioexception.
> {code:java}
> // exception information
> java.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not existjava.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not exist at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
> {code}
> When I update the tempFileName from '.distcp.tmp.XXX' in
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#getTempFile to
> '.distcp.tmp.XXX', the question one is gone.
> The targetPath can be renamed when only the path of uri exists in
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#doCopy.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17742) DistCp: distcp fail when copying to ftp filesystem
[
https://issues.apache.org/jira/browse/HADOOP-17742?focusedWorklogId=605215&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605215
]
ASF GitHub Bot logged work on HADOOP-17742:
---
Author: ASF GitHub Bot
Created on: 02/Jun/21 13:58
Start Date: 02/Jun/21 13:58
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3067:
URL: https://github.com/apache/hadoop/pull/3067#issuecomment-853051861
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 34s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 17s | | trunk passed |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 49s | | trunk passed |
| +1 :green_heart: | shadedclient | 13m 56s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | compile | 0m 22s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 0m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 25s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 48s | | the patch passed |
| +1 :green_heart: | shadedclient | 13m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 20m 41s | | hadoop-distcp in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 93m 50s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3067/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3067 |
| JIRA Issue | HADOOP-17742 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux ba7d4d3a0ded 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28
05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fe796ebc9e484f225997ba15b9905447072e26e4 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3067/1/testReport/ |
| Max. process+thread count | 546 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3067/1/c
[jira] [Commented] (HADOOP-17742) DistCp: distcp fail when copying to ftp filesystem
[
https://issues.apache.org/jira/browse/HADOOP-17742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17355749#comment-17355749
]
Hadoop QA commented on HADOOP-17742:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
34s{color} | | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} codespell {color} | {color:blue} 0m
1s{color} | | {color:blue} codespell was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | | {color:green} The patch does not contain any @author tags.
{color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | | {color:red} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 35m
17s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 0m
49s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 56s{color} | | {color:green} branch has no errors when building and
testing our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green} 0m
0s{color} | | {color:green} The patch has no blanks issues. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 0m
48s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 45s{color} | | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} || ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 20m
41s{color} | | {color:green} hadoop-distcp in the pa
[GitHub] [hadoop] hadoop-yetus commented on pull request #3067: HADOOP-17742. fix distcp fail when copying to ftp filesystem
hadoop-yetus commented on pull request #3067:
URL: https://github.com/apache/hadoop/pull/3067#issuecomment-853051861
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 34s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 17s | | trunk passed |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 49s | | trunk passed |
| +1 :green_heart: | shadedclient | 13m 56s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | compile | 0m 22s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 0m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 25s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 48s | | the patch passed |
| +1 :green_heart: | shadedclient | 13m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 20m 41s | | hadoop-distcp in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 93m 50s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3067/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3067 |
| JIRA Issue | HADOOP-17742 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux ba7d4d3a0ded 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28
05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fe796ebc9e484f225997ba15b9905447072e26e4 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3067/1/testReport/ |
| Max. process+thread count | 546 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3067/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at
[jira] [Resolved] (HADOOP-17742) DistCp: distcp fail when copying to ftp filesystem
[
https://issues.apache.org/jira/browse/HADOOP-17742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jerry resolved HADOOP-17742.
Resolution: Fixed
> DistCp: distcp fail when copying to ftp filesystem
> --
>
> Key: HADOOP-17742
> URL: https://issues.apache.org/jira/browse/HADOOP-17742
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 3.3.0
>Reporter: jerry
>Assignee: jerry
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When I copy file from a ftp filesystem to other ftp filesystem by distCp,
> there are two problems. Firstly, the tmp target path can't be found in the
> exception information. After my test, it was caused by the comma. Then when
> the uri of target path with schema, host, port can't be rename because of the
> ioexception.
> {code:java}
> // exception information
> java.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not existjava.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not exist at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
> {code}
> When I update the tempFileName from '.distcp.tmp.XXX' in
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#getTempFile to
> '.distcp.tmp.XXX', the question one is gone.
> The targetPath can be renamed when only the path of uri exists in
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#doCopy.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HADOOP-17409: --- Assignee: Mehakmeet Singh > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Mehakmeet Singh >Priority: Major > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-17409) Remove S3Guard - no longer needed
[ https://issues.apache.org/jira/browse/HADOOP-17409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17331287#comment-17331287 ] Steve Loughran edited comment on HADOOP-17409 at 6/2/21, 1:49 PM: -- IT's all in 3.3.1 just not needed any more. I plan to worry about how to cut it from the code after the auditing PR HADOOP-17511 is in Keeping in s3guard complicates our life -too many branches in the code of with/without s3guard, and doubles the amount of test runs needed. IT also complicates any attempt to do maintenance refactoring (see https://github.com/steveloughran/engineering-proposals/blob/trunk/refactoring-s3a.md ) It's going to take care to remove. I'm thinking of having a separate apache branch where we remove different pieces one by one # dynamodb (just have local for testing) # the various tools (import, prune, fsck and their tests) # Reading from s3guard in list/getFileStatus etc. Will need to tweak tests which fail when s3guard is on-or just disable them # writing to it. # all S3guard only tests # All parameterized tests with guarded/raw phases was (Author: ste...@apache.org): IT's all in 3.3.1 just not needed any more. I plan to worry about how to cut it from the code after the auditing PR HADOOP-17511 is in Keeping in s3guard complicates our life -too many branches in the code of with/without s3guard, and doubles the amount of test runs needed. IT also complicates any attempt to do maintenance refactoring (see https://github.com/steveloughran/engineering-proposals/blob/trunk/refactoring-s3a.md ) It's going to take care to remove. I'm thinking of having a separate apache branch where we remove different pieces one by one # dynamodb (just have local for testing) # the various tools (import, prune, fsck and their tests) # Reading from s3guard in list/getFileStatus etc. Will need to tweak tests which fail when s3guard is on-or just disable them # writing to it. # all S3guard only tests # All parameterized tests with guarded/raw phases # u > Remove S3Guard - no longer needed > - > > Key: HADOOP-17409 > URL: https://issues.apache.org/jira/browse/HADOOP-17409 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Priority: Major > > With Consistent S3, S3Guard is superfluous. > stop developing it and wean people off it as soon as they can. > Then we can worry about what to do in the code. It has gradually insinuated > its way through the layers, especially things like multi-object delete > handling (see HADOOP-17244). Things would be a lot simpler without it -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2894: HDFS-15966. Empty the statistical parameters when emptying the redundant queue
hadoop-yetus commented on pull request #2894:
URL: https://github.com/apache/hadoop/pull/2894#issuecomment-853011002
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 46s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 10s | | trunk passed |
| +1 :green_heart: | compile | 1m 27s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 20s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 3s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 57s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 27s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 27s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 47s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 20s | | the patch passed |
| +1 :green_heart: | compile | 1m 22s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 22s | | the patch passed |
| +1 :green_heart: | compile | 1m 13s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 1m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 55s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 20s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 50s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 27s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 31s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 14s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 240m 46s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 336m 12s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2894/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2894 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 58bf96c83a7c 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c343a52432d864e63f39bfde136d8a2cbe89abcb |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2894/5/testReport/ |
| Max. process+thread count | 3195 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2894/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hadoop] ferhui commented on pull request #3065: HDFS-13671. Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet
ferhui commented on pull request #3065: URL: https://github.com/apache/hadoop/pull/3065#issuecomment-853004590 Failed tests pass locally. Now it is already for review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-17742) DistCp: distcp fail when copying to ftp filesystem
[
https://issues.apache.org/jira/browse/HADOOP-17742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HADOOP-17742 started by jerry.
--
> DistCp: distcp fail when copying to ftp filesystem
> --
>
> Key: HADOOP-17742
> URL: https://issues.apache.org/jira/browse/HADOOP-17742
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 3.3.0
>Reporter: jerry
>Assignee: jerry
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When I copy file from a ftp filesystem to other ftp filesystem by distCp,
> there are two problems. Firstly, the tmp target path can't be found in the
> exception information. After my test, it was caused by the comma. Then when
> the uri of target path with schema, host, port can't be rename because of the
> ioexception.
> {code:java}
> // exception information
> java.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not existjava.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not exist at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
> {code}
> When I update the tempFileName from '.distcp.tmp.XXX' in
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#getTempFile to
> '.distcp.tmp.XXX', the question one is gone.
> The targetPath can be renamed when only the path of uri exists in
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#doCopy.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ferhui commented on a change in pull request #3065: HDFS-13671. Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet
ferhui commented on a change in pull request #3065:
URL: https://github.com/apache/hadoop/pull/3065#discussion_r643905689
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
##
@@ -1996,7 +1996,11 @@ private void metaSave(PrintWriter out) {
LightWeightHashSet openFileIds = new LightWeightHashSet<>();
for (DatanodeDescriptor dataNode :
blockManager.getDatanodeManager().getDatanodes()) {
- for (long ucFileId : dataNode.getLeavingServiceStatus().getOpenFiles()) {
+ // Sort open files
+ LightWeightHashSet dnOpenFiles =
dataNode.getLeavingServiceStatus().getOpenFiles();
Review comment:
checkstyle
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17742) DistCp: distcp fail when copying to ftp filesystem
[
https://issues.apache.org/jira/browse/HADOOP-17742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HADOOP-17742:
Labels: pull-request-available (was: )
> DistCp: distcp fail when copying to ftp filesystem
> --
>
> Key: HADOOP-17742
> URL: https://issues.apache.org/jira/browse/HADOOP-17742
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 3.3.0
>Reporter: jerry
>Assignee: jerry
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When I copy file from a ftp filesystem to other ftp filesystem by distCp,
> there are two problems. Firstly, the tmp target path can't be found in the
> exception information. After my test, it was caused by the comma. Then when
> the uri of target path with schema, host, port can't be rename because of the
> ioexception.
> {code:java}
> // exception information
> java.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not existjava.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not exist at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
> {code}
> When I update the tempFileName from '.distcp.tmp.XXX' in
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#getTempFile to
> '.distcp.tmp.XXX', the question one is gone.
> The targetPath can be renamed when only the path of uri exists in
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#doCopy.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17742) DistCp: distcp fail when copying to ftp filesystem
[
https://issues.apache.org/jira/browse/HADOOP-17742?focusedWorklogId=605149&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605149
]
ASF GitHub Bot logged work on HADOOP-17742:
---
Author: ASF GitHub Bot
Created on: 02/Jun/21 12:23
Start Date: 02/Jun/21 12:23
Worklog Time Spent: 10m
Work Description: zhaomin1423 opened a new pull request #3067:
URL: https://github.com/apache/hadoop/pull/3067
There is more Detailed information.
https://issues.apache.org/jira/browse/HADOOP-17742#
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 605149)
Remaining Estimate: 0h
Time Spent: 10m
> DistCp: distcp fail when copying to ftp filesystem
> --
>
> Key: HADOOP-17742
> URL: https://issues.apache.org/jira/browse/HADOOP-17742
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 3.3.0
>Reporter: jerry
>Assignee: jerry
>Priority: Blocker
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When I copy file from a ftp filesystem to other ftp filesystem by distCp,
> there are two problems. Firstly, the tmp target path can't be found in the
> exception information. After my test, it was caused by the comma. Then when
> the uri of target path with schema, host, port can't be rename because of the
> ioexception.
> {code:java}
> // exception information
> java.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not existjava.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not exist at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
> {code}
> When I update the tempFileName from '.distcp.tmp.XXX' in
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#getTempFile to
> '.distcp.tmp.XXX', the question one is gone.
> The targetPath can be renamed when only the path of uri exists in
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#doCopy.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] zhaomin1423 opened a new pull request #3067: [HADOOP-17742] fix distcp fail when copying to ftp filesystem
zhaomin1423 opened a new pull request #3067: URL: https://github.com/apache/hadoop/pull/3067 There is more Detailed information. https://issues.apache.org/jira/browse/HADOOP-17742# -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17742) DistCp: distcp fail when copying to ftp filesystem
[
https://issues.apache.org/jira/browse/HADOOP-17742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jerry updated HADOOP-17742:
---
Description:
When I copy file from a ftp filesystem to other ftp filesystem by distCp, there
are two problems. Firstly, the tmp target path can't be found in the exception
information. After my test, it was caused by the comma. Then when the uri of
target path with schema, host, port can't be rename because of the ioexception.
{code:java}
// exception information
java.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not existjava.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not exist at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
{code}
When I update the tempFileName from '.distcp.tmp.XXX' in
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#getTempFile to
'.distcp.tmp.XXX', the question one is gone.
The targetPath can be renamed when only the path of uri exists.
was:
When I copy file from a ftp filesystem to other ftp filesystem by distCp, there
are two problems. Firstly, the tmp target path can't be found in the exception
information. After my test, it was caused by the comma. Then when the uri of
target path with schema, host, port can't be rename because of the ioexception.
{code:java}
// exception information
java.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not existjava.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not exist at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
{code}
When I update the tempFileName from '.distcp.tmp.XXX' in
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#getTempFile to
'.distcp.tmp.XXX', the question one is gone.
> DistCp: distcp fail when copying to ftp filesystem
> --
>
> Key: HADOOP-17742
> URL: https://issues.apache.org/jira/browse/HADOOP-17742
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 3.3.0
>Reporter: jerry
>Assignee: jerry
>Priority: Blocker
>
> When I copy file from a ftp filesystem to other ftp filesystem by distCp,
> there are two problems. Firstly, the tmp target path can't be found in the
> exception information. After my test, it was caused by the comma. Then when
> the uri of target path with schema, host, port can't be rename because of the
> ioexception.
> {code:java}
> // exception information
> java.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not existjava.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not exist at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
> at
> org.apache.hadoop.tools.mapred.
[jira] [Updated] (HADOOP-17742) DistCp: distcp fail when copying to ftp filesystem
[
https://issues.apache.org/jira/browse/HADOOP-17742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jerry updated HADOOP-17742:
---
Description:
When I copy file from a ftp filesystem to other ftp filesystem by distCp, there
are two problems. Firstly, the tmp target path can't be found in the exception
information. After my test, it was caused by the comma. Then when the uri of
target path with schema, host, port can't be rename because of the ioexception.
{code:java}
// exception information
java.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not existjava.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not exist at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
{code}
When I update the tempFileName from '.distcp.tmp.XXX' in
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#getTempFile to
'.distcp.tmp.XXX', the question one is gone.
The targetPath can be renamed when only the path of uri exists in
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#doCopy.
was:
When I copy file from a ftp filesystem to other ftp filesystem by distCp, there
are two problems. Firstly, the tmp target path can't be found in the exception
information. After my test, it was caused by the comma. Then when the uri of
target path with schema, host, port can't be rename because of the ioexception.
{code:java}
// exception information
java.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not existjava.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not exist at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
{code}
When I update the tempFileName from '.distcp.tmp.XXX' in
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#getTempFile to
'.distcp.tmp.XXX', the question one is gone.
The targetPath can be renamed when only the path of uri exists.
> DistCp: distcp fail when copying to ftp filesystem
> --
>
> Key: HADOOP-17742
> URL: https://issues.apache.org/jira/browse/HADOOP-17742
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 3.3.0
>Reporter: jerry
>Assignee: jerry
>Priority: Blocker
>
> When I copy file from a ftp filesystem to other ftp filesystem by distCp,
> there are two problems. Firstly, the tmp target path can't be found in the
> exception information. After my test, it was caused by the comma. Then when
> the uri of target path with schema, host, port can't be rename because of the
> ioexception.
> {code:java}
> // exception information
> java.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not existjava.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not exist at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
> org.apache.hadoop.to
[GitHub] [hadoop] hadoop-yetus commented on pull request #3065: HDFS-13671. Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet
hadoop-yetus commented on pull request #3065:
URL: https://github.com/apache/hadoop/pull/3065#issuecomment-852973544
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 18 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 9s | | trunk passed |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 8s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 55s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 23s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 17s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | cc | 1m 18s | | the patch passed |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 8s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | cc | 1m 8s | | the patch passed |
| +1 :green_heart: | javac | 1m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 2s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3065/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 1338 unchanged
- 13 fixed = 1340 total (was 1351) |
| +1 :green_heart: | mvnsite | 1m 16s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 47s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 21s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 55s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 354m 59s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3065/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| -1 :x: | asflicense | 0m 39s |
[/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3065/2/artifact/out/results-asflicense.txt)
| The patch generated 2 ASF License warnings. |
| | | 447m 0s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList |
| |
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3065/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3065 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell cc buflint bufcompat
xml |
| uname | Linux cdb05ad52f99 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28
05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh
[jira] [Updated] (HADOOP-17742) DistCp: distcp fail when copying to ftp filesystem
[
https://issues.apache.org/jira/browse/HADOOP-17742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jerry updated HADOOP-17742:
---
Description:
When I copy file from a ftp filesystem to other ftp filesystem by distCp, there
are two problems. Firstly, the tmp target path can't be found in the exception
information. After my test, it was caused by the comma. Then when the uri of
target path with schema, host, port can't be rename because of the ioexception.
{code:java}
// exception information
java.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not existjava.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not exist at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
{code}
When I update the tempFileName from '.distcp.tmp.XXX' in
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#getTempFile to
'.distcp.tmp.XXX', the question one is gone.
was:
When I copy file from a ftp filesystem to other ftp filesystem by distCp, there
are two question. Firstly, the tmp target path can't be found in the exception
information. After my test, it was caused by the comma. Then when the uri of
target path with schema, host, port can't be rename because of the ioexception.
{code:java}
// exception information
java.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not existjava.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not exist at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
{code}
When I update the tempFileName from '.distcp.tmp.XXX' in
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#getTempFile to
'.distcp.tmp.XXX', the question one is gone.
> DistCp: distcp fail when copying to ftp filesystem
> --
>
> Key: HADOOP-17742
> URL: https://issues.apache.org/jira/browse/HADOOP-17742
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 3.3.0
>Reporter: jerry
>Assignee: jerry
>Priority: Blocker
>
> When I copy file from a ftp filesystem to other ftp filesystem by distCp,
> there are two problems. Firstly, the tmp target path can't be found in the
> exception information. After my test, it was caused by the comma. Then when
> the uri of target path with schema, host, port can't be rename because of the
> ioexception.
> {code:java}
> // exception information
> java.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not existjava.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not exist at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:1
[jira] [Updated] (HADOOP-17742) DistCp: distcp fail when copying to ftp filesystem
[
https://issues.apache.org/jira/browse/HADOOP-17742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jerry updated HADOOP-17742:
---
Description:
{code:java}
When I copy file from a ftp filesystem to other ftp filesystem by distCp, there
are two question. Firstly, the tmp target path can't be found in the exception
information. After my test, it was caused by the comma. Then when the uri of
target path with schema, host, port can't be rename because of the ioexception.
// exception information
java.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not existjava.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not exist at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
{code}
When I update the tempFileName from '.distcp.tmp.XXX' in
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#getTempFile to
'.distcp.tmp.XXX', the question one is gone.
was:
{code:java}
When I copy file from a ftp filesystem to other ftp filesystem by distCp, the
tmp target path can't be found in the exception information. After my test, it
was caused by the comma. And when the uri of target path with schema, host,
port can't be rename because of the ioexception.
// exception information
java.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not existjava.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not exist at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
{code}
> DistCp: distcp fail when copying to ftp filesystem
> --
>
> Key: HADOOP-17742
> URL: https://issues.apache.org/jira/browse/HADOOP-17742
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 3.3.0
>Reporter: jerry
>Assignee: jerry
>Priority: Blocker
>
> {code:java}
> When I copy file from a ftp filesystem to other ftp filesystem by distCp,
> there are two question. Firstly, the tmp target path can't be found in the
> exception information. After my test, it was caused by the comma. Then when
> the uri of target path with schema, host, port can't be rename because of the
> ioexception.
> // exception information
> java.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not existjava.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not exist at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> at
[jira] [Updated] (HADOOP-17742) DistCp: distcp fail when copying to ftp filesystem
[
https://issues.apache.org/jira/browse/HADOOP-17742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jerry updated HADOOP-17742:
---
Description:
When I copy file from a ftp filesystem to other ftp filesystem by distCp, there
are two question. Firstly, the tmp target path can't be found in the exception
information. After my test, it was caused by the comma. Then when the uri of
target path with schema, host, port can't be rename because of the ioexception.
{code:java}
// exception information
java.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not existjava.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not exist at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
{code}
When I update the tempFileName from '.distcp.tmp.XXX' in
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#getTempFile to
'.distcp.tmp.XXX', the question one is gone.
was:
{code:java}
When I copy file from a ftp filesystem to other ftp filesystem by distCp, there
are two question. Firstly, the tmp target path can't be found in the exception
information. After my test, it was caused by the comma. Then when the uri of
target path with schema, host, port can't be rename because of the ioexception.
// exception information
java.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not existjava.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not exist at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
{code}
When I update the tempFileName from '.distcp.tmp.XXX' in
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand#getTempFile to
'.distcp.tmp.XXX', the question one is gone.
> DistCp: distcp fail when copying to ftp filesystem
> --
>
> Key: HADOOP-17742
> URL: https://issues.apache.org/jira/browse/HADOOP-17742
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 3.3.0
>Reporter: jerry
>Assignee: jerry
>Priority: Blocker
>
> When I copy file from a ftp filesystem to other ftp filesystem by distCp,
> there are two question. Firstly, the tmp target path can't be found in the
> exception information. After my test, it was caused by the comma. Then when
> the uri of target path with schema, host, port can't be rename because of the
> ioexception.
> {code:java}
> // exception information
> java.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not existjava.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not exist at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:
[jira] [Updated] (HADOOP-17742) DistCp: distcp fail when copying to ftp filesystem
[
https://issues.apache.org/jira/browse/HADOOP-17742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jerry updated HADOOP-17742:
---
Description:
{code:java}
When I copy file from a ftp filesystem to other ftp filesystem by distCp, the
tmp target path can't be found in the exception information. After my test, it
was caused by the comma. And when the uri of target path with schema, host,
port can't be rename because of the ioexception.
// exception information
java.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not existjava.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not exist at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
{code}
was:
{code:java}
When I copy file from a ftp filesystem to other ftp filesystem by distCp, the
tmp target path can't be found. After my test, it was caused by the comma. And
when the uri of target path with schema, host, port can't be rename because of
the ioexception.
// exception information
java.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not existjava.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not exist at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
{code}
> DistCp: distcp fail when copying to ftp filesystem
> --
>
> Key: HADOOP-17742
> URL: https://issues.apache.org/jira/browse/HADOOP-17742
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 3.3.0
>Reporter: jerry
>Assignee: jerry
>Priority: Blocker
>
> {code:java}
> When I copy file from a ftp filesystem to other ftp filesystem by distCp, the
> tmp target path can't be found in the exception information. After my test,
> it was caused by the comma. And when the uri of target path with schema,
> host, port can't be rename because of the ioexception.
> // exception information
> java.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not existjava.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not exist at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hado
[GitHub] [hadoop] virajjasani commented on pull request #2998: HDFS-16016. BPServiceActor to provide new thread to handle IBR
virajjasani commented on pull request #2998: URL: https://github.com/apache/hadoop/pull/2998#issuecomment-852966245 FYI @liuml07 if you get some bandwidth. Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17742) DistCp: distcp fail when copying to ftp filesystem
[
https://issues.apache.org/jira/browse/HADOOP-17742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jerry updated HADOOP-17742:
---
Description:
{code:java}
When I copy file from a ftp filesystem to other ftp filesystem by distCp, the
tmp target path can't be found. After my test, it was caused by the comma. And
when the uri of target path with schema, host, port can't be rename because of
the ioexception.
// exception information
java.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not existjava.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not exist at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
{code}
was:
{code:java}
When I copy file from a ftp filesystem to other ftp filesystem by distCp, the
tmp target path can't be found. After my test, it was caused by the comma. And
when the uri of target path with schema, host, port can't be rename because of
the ioexception.
// exception information
java.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not existjava.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not exist at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
// exception information when rename
Exception in thread "main" java.io.IOException: Cannot rename source:
/ndi-0111/2/distcp.tmp.attempt_local1899244355_0001_m_00_0.1622622033323 to
ftp://59.111.93.226:60021/ndi-0111/2/test6.txt -only same directory renames are
supportedException in thread "main" java.io.IOException: Cannot rename source:
/ndi-0111/2/distcp.tmp.attempt_local1899244355_0001_m_00_0.1622622033323 to
ftp://59.111.93.226:60021/ndi-0111/2/test6.txt -only same directory renames are
supported at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:674) at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:613)
{code}
> DistCp: distcp fail when copying to ftp filesystem
> --
>
> Key: HADOOP-17742
> URL: https://issues.apache.org/jira/browse/HADOOP-17742
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 3.3.0
>Reporter: jerry
>Assignee: jerry
>Priority: Blocker
>
> {code:java}
> When I copy file from a ftp filesystem to other ftp filesystem by distCp, the
> tmp target path can't be found. After my test, it was caused by the comma.
> And when the uri of target path with schema, host, port can't be rename
> because of the ioexception.
> // exception information
> java.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not existjava.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not exist at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileC
[jira] [Updated] (HADOOP-17742) DistCp: distcp fail when copying to ftp filesystem
[
https://issues.apache.org/jira/browse/HADOOP-17742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jerry updated HADOOP-17742:
---
Description:
{code:java}
When I copy file from a ftp filesystem to other ftp filesystem by distCp, the
tmp target path can't be found. After my test, it was caused by the comma. And
when the uri of target path with schema, host, port can't be rename because of
the ioexception.
// exception information
java.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not existjava.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not exist at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
// exception information when rename
Exception in thread "main" java.io.IOException: Cannot rename source:
/ndi-0111/2/distcp.tmp.attempt_local1899244355_0001_m_00_0.1622622033323 to
ftp://59.111.93.226:60021/ndi-0111/2/test6.txt -only same directory renames are
supportedException in thread "main" java.io.IOException: Cannot rename source:
/ndi-0111/2/distcp.tmp.attempt_local1899244355_0001_m_00_0.1622622033323 to
ftp://59.111.93.226:60021/ndi-0111/2/test6.txt -only same directory renames are
supported at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:674) at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:613)
{code}
was:
{code:java}
When I copy file from a ftp filesystem to other ftp filesystem, the tmp target
path can't be found. After my test, it was caused by the comma. And when the
uri of target path with schema, host, port can't be rename.
// exception information
java.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not existjava.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not exist at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
{code}
> DistCp: distcp fail when copying to ftp filesystem
> --
>
> Key: HADOOP-17742
> URL: https://issues.apache.org/jira/browse/HADOOP-17742
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 3.3.0
>Reporter: jerry
>Assignee: jerry
>Priority: Blocker
>
> {code:java}
> When I copy file from a ftp filesystem to other ftp filesystem by distCp, the
> tmp target path can't be found. After my test, it was caused by the comma.
> And when the uri of target path with schema, host, port can't be rename
> because of the ioexception.
> // exception information
> java.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not existjava.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not exist at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
> at
> org.apac
[jira] [Updated] (HADOOP-17742) DistCp: distcp fail when copying to ftp filesystem
[
https://issues.apache.org/jira/browse/HADOOP-17742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jerry updated HADOOP-17742:
---
Description:
{code:java}
When I copy file from a ftp filesystem to other ftp filesystem, the tmp target
path can't be found. After my test, it was caused by the comma. And when the
uri of target path with schema, host, port can't be rename.
// exception information
java.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not existjava.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not exist at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
{code}
was:
{code:java}
When I copy file from a ftp filesystem to other ftp filesystem, the tmp target
path can't be found.
// exception information
java.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not existjava.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not exist at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
{code}
> DistCp: distcp fail when copying to ftp filesystem
> --
>
> Key: HADOOP-17742
> URL: https://issues.apache.org/jira/browse/HADOOP-17742
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 3.3.0
>Reporter: jerry
>Assignee: jerry
>Priority: Blocker
>
> {code:java}
> When I copy file from a ftp filesystem to other ftp filesystem, the tmp
> target path can't be found. After my test, it was caused by the comma. And
> when the uri of target path with schema, host, port can't be rename.
> // exception information
> java.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not existjava.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not exist at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17742) DistCp: distcp fail when copying to ftp filesystem
[
https://issues.apache.org/jira/browse/HADOOP-17742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jerry updated HADOOP-17742:
---
Description:
{code:java}
When I copy file from a ftp filesystem to other ftp filesystem, the tmp target
path can't be found.
// exception information
java.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not existjava.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not exist at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
{code}
was:
{code:java}
//代码占位符
java.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not existjava.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not exist at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
{code}
> DistCp: distcp fail when copying to ftp filesystem
> --
>
> Key: HADOOP-17742
> URL: https://issues.apache.org/jira/browse/HADOOP-17742
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 3.3.0
>Reporter: jerry
>Priority: Blocker
>
> {code:java}
> When I copy file from a ftp filesystem to other ftp filesystem, the tmp
> target path can't be found.
> // exception information
> java.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not existjava.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not exist at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-17742) DistCp: distcp fail when copying to ftp filesystem
[
https://issues.apache.org/jira/browse/HADOOP-17742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jerry reassigned HADOOP-17742:
--
Assignee: jerry
> DistCp: distcp fail when copying to ftp filesystem
> --
>
> Key: HADOOP-17742
> URL: https://issues.apache.org/jira/browse/HADOOP-17742
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 3.3.0
>Reporter: jerry
>Assignee: jerry
>Priority: Blocker
>
> {code:java}
> When I copy file from a ftp filesystem to other ftp filesystem, the tmp
> target path can't be found.
> // exception information
> java.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not existjava.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not exist at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17742) DistCp: distcp fail when copying to ftp filesystem
[
https://issues.apache.org/jira/browse/HADOOP-17742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jerry updated HADOOP-17742:
---
Description:
{code:java}
//代码占位符
java.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not existjava.io.FileNotFoundException: Source path
ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
does not exist at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
{code}
> DistCp: distcp fail when copying to ftp filesystem
> --
>
> Key: HADOOP-17742
> URL: https://issues.apache.org/jira/browse/HADOOP-17742
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 3.3.0
>Reporter: jerry
>Priority: Blocker
>
> {code:java}
> //代码占位符
> java.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not existjava.io.FileNotFoundException: Source path
> ftp://hadoop336.photo.163.org:21/ndi-0111/56598489-3de3-4585-8426-c03435640481/.distcp.tmp.attempt_local344388648_0001_m_00_0.1622634094600
> does not exist at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:668) at
> org.apache.hadoop.fs.ftp.FTPFileSystem.rename(FTPFileSystem.java:626) at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:155)
> at
> org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115)
> at
> org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
> at
> org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17544) Mark KeyProvider as Stable
[ https://issues.apache.org/jira/browse/HADOOP-17544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17355646#comment-17355646 ] Akira Ajisaka commented on HADOOP-17544: Hi [~shv], what do you think? > Mark KeyProvider as Stable > -- > > Key: HADOOP-17544 > URL: https://issues.apache.org/jira/browse/HADOOP-17544 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Now, o.a.h.crypto.key.KeyProvider.java is marked Public and Unstable. I think > the class is very stable, and it should be annotated as Stable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17544) Mark KeyProvider as Stable
[
https://issues.apache.org/jira/browse/HADOOP-17544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17355645#comment-17355645
]
Akira Ajisaka commented on HADOOP-17544:
FYI: LinkedIn used the KeyProvider interface to integrate the company's
internal service.
[https://engineering.linkedin.com/blog/2021/the-exabyte-club--linkedin-s-journey-of-scaling-the-hadoop-distr]
{quote}LinkedIn has its own key management service, LiKMS, which is the only
service certified and approved for managing cryptographic keys and secrets
internally. We used pluggable interfaces such as KeyProvider supported by HDFS
to integrate LiKMS with transparent encryption at rest.
{quote}
> Mark KeyProvider as Stable
> --
>
> Key: HADOOP-17544
> URL: https://issues.apache.org/jira/browse/HADOOP-17544
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Now, o.a.h.crypto.key.KeyProvider.java is marked Public and Unstable. I think
> the class is very stable, and it should be annotated as Stable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2894: HDFS-15966. Empty the statistical parameters when emptying the redundant queue
hadoop-yetus commented on pull request #2894:
URL: https://github.com/apache/hadoop/pull/2894#issuecomment-852941309
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 58s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 39s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 21s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 56s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 28s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 4s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 26s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 10s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 1m 10s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 12s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 46s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 19s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 11s | | the patch passed |
| +1 :green_heart: | shadedclient | 17m 26s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 470m 11s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2894/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 561m 1s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.fs.viewfs.TestViewFSOverloadSchemeWithMountTableConfigInHDFS |
| | hadoop.hdfs.web.TestWebHdfsFileSystemContract |
| |
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor |
| | hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.TestSnapshotCommands |
| | hadoop.hdfs.TestHDFSFileSystemContract |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
| | hadoop.hdfs.TestViewDistributedFileSystemWithMountLinks |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| |
hadoop.fs.viewfs.TestViewFileSystemOverloadSchemeHdfsFileSystemContract |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2894/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2894 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux c1f192bcbff2 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
[jira] [Created] (HADOOP-17742) DistCp: distcp fail when copying to ftp filesystem
jerry created HADOOP-17742: -- Summary: DistCp: distcp fail when copying to ftp filesystem Key: HADOOP-17742 URL: https://issues.apache.org/jira/browse/HADOOP-17742 Project: Hadoop Common Issue Type: Bug Components: tools/distcp Affects Versions: 3.3.0 Reporter: jerry -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17590) ABFS: Introduce Lease Operations with Append to provide single writer semantics
[
https://issues.apache.org/jira/browse/HADOOP-17590?focusedWorklogId=605116&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605116
]
ASF GitHub Bot logged work on HADOOP-17590:
---
Author: ASF GitHub Bot
Created on: 02/Jun/21 11:06
Start Date: 02/Jun/21 11:06
Worklog Time Spent: 10m
Work Description: snvijaya commented on a change in pull request #3026:
URL: https://github.com/apache/hadoop/pull/3026#discussion_r643841953
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##
@@ -521,12 +523,14 @@ public OutputStream createFile(final Path path,
isNamespaceEnabled ? getOctalNotation(permission) : null,
isNamespaceEnabled ? getOctalNotation(umask) : null,
isAppendBlob,
-null);
+null,
+lease);
}
perfInfo.registerResult(op.getResult()).registerSuccess(true);
- AbfsLease lease = maybeCreateLease(relativePath);
-
+ if (lease == null) {
Review comment:
Acquiring lease along with create/append enables atomic fetch of lease
with the store operation and prevent any parallel writers. If configured for
infinite lease, the path creation will happen without bundled lease acquiry and
rely on later acquireLease API call in AbfsLease. Infinite lease flow should
purely be on acquireLease API only if fs.azure.write.enforceLease is off.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 605116)
Time Spent: 2h 40m (was: 2.5h)
> ABFS: Introduce Lease Operations with Append to provide single writer
> semantics
> ---
>
> Key: HADOOP-17590
> URL: https://issues.apache.org/jira/browse/HADOOP-17590
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Sneha Varma
>Assignee: Sneha Varma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> The lease operations will be introduced as part of Append, Flush to ensure
> the single writer semantics.
>
> Details:
> Acquire Lease will be introduced in Create, Auto-Renew, Acquire will be added
> to Append & Release, Auto-Renew, Acquire in Flush.
>
> Duration the creation of the file the lease will be acquired, as part of
> appends the lease will be auto-renewed & the lease can be released as part of
> flush.
>
> By default the lease duration will be of 60 seconds.
> "fs.azure.write.enforcelease" & "fs.azure.write.lease.duration" two configs
> will be introduced.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #3026: HADOOP-17590 ABFS: Introduce Lease Operations with Append to provide single writer semantics
snvijaya commented on a change in pull request #3026:
URL: https://github.com/apache/hadoop/pull/3026#discussion_r643841953
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##
@@ -521,12 +523,14 @@ public OutputStream createFile(final Path path,
isNamespaceEnabled ? getOctalNotation(permission) : null,
isNamespaceEnabled ? getOctalNotation(umask) : null,
isAppendBlob,
-null);
+null,
+lease);
}
perfInfo.registerResult(op.getResult()).registerSuccess(true);
- AbfsLease lease = maybeCreateLease(relativePath);
-
+ if (lease == null) {
Review comment:
Acquiring lease along with create/append enables atomic fetch of lease
with the store operation and prevent any parallel writers. If configured for
infinite lease, the path creation will happen without bundled lease acquiry and
rely on later acquireLease API call in AbfsLease. Infinite lease flow should
purely be on acquireLease API only if fs.azure.write.enforceLease is off.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2894: HDFS-15966. Empty the statistical parameters when emptying the redundant queue
hadoop-yetus commented on pull request #2894:
URL: https://github.com/apache/hadoop/pull/2894#issuecomment-852932585
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 19s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 13s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 55s | | trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 26s | | trunk passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 9s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 2s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 52s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 12s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 46s | | the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 22s | | the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 22s | | the patch passed |
| +1 :green_heart: | shadedclient | 17m 9s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 466m 1s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2894/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 48s | | The patch does not
generate ASF License warnings. |
| | | 556m 25s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.fs.viewfs.TestViewFSOverloadSchemeWithMountTableConfigInHDFS |
| | hadoop.hdfs.web.TestWebHdfsFileSystemContract |
| |
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor |
| | hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand |
| | hadoop.hdfs.TestViewDistributedFileSystemContract |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.TestSnapshotCommands |
| | hadoop.hdfs.TestHDFSFileSystemContract |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| |
hadoop.fs.viewfs.TestViewFileSystemOverloadSchemeHdfsFileSystemContract |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2894/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2894 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux d16502d18367 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| gi
[jira] [Work logged] (HADOOP-17590) ABFS: Introduce Lease Operations with Append to provide single writer semantics
[ https://issues.apache.org/jira/browse/HADOOP-17590?focusedWorklogId=605110&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605110 ] ASF GitHub Bot logged work on HADOOP-17590: --- Author: ASF GitHub Bot Created on: 02/Jun/21 10:52 Start Date: 02/Jun/21 10:52 Worklog Time Spent: 10m Work Description: snehavarma commented on a change in pull request #3026: URL: https://github.com/apache/hadoop/pull/3026#discussion_r643844817 ## File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java ## @@ -506,13 +506,15 @@ public OutputStream createFile(final Path path, triggerConditionalCreateOverwrite = true; } + AbfsLease lease = maybeCreateFiniteLease(relativePath, isNamespaceEnabled); Review comment: No it will not, it returns a null object if path qualifies for infinite lease -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 605110) Time Spent: 2.5h (was: 2h 20m) > ABFS: Introduce Lease Operations with Append to provide single writer > semantics > --- > > Key: HADOOP-17590 > URL: https://issues.apache.org/jira/browse/HADOOP-17590 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sneha Varma >Assignee: Sneha Varma >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > The lease operations will be introduced as part of Append, Flush to ensure > the single writer semantics. > > Details: > Acquire Lease will be introduced in Create, Auto-Renew, Acquire will be added > to Append & Release, Auto-Renew, Acquire in Flush. > > Duration the creation of the file the lease will be acquired, as part of > appends the lease will be auto-renewed & the lease can be released as part of > flush. > > By default the lease duration will be of 60 seconds. > "fs.azure.write.enforcelease" & "fs.azure.write.lease.duration" two configs > will be introduced. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snehavarma commented on a change in pull request #3026: HADOOP-17590 ABFS: Introduce Lease Operations with Append to provide single writer semantics
snehavarma commented on a change in pull request #3026: URL: https://github.com/apache/hadoop/pull/3026#discussion_r643844817 ## File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java ## @@ -506,13 +506,15 @@ public OutputStream createFile(final Path path, triggerConditionalCreateOverwrite = true; } + AbfsLease lease = maybeCreateFiniteLease(relativePath, isNamespaceEnabled); Review comment: No it will not, it returns a null object if path qualifies for infinite lease -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17590) ABFS: Introduce Lease Operations with Append to provide single writer semantics
[
https://issues.apache.org/jira/browse/HADOOP-17590?focusedWorklogId=605107&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605107
]
ASF GitHub Bot logged work on HADOOP-17590:
---
Author: ASF GitHub Bot
Created on: 02/Jun/21 10:49
Start Date: 02/Jun/21 10:49
Worklog Time Spent: 10m
Work Description: snvijaya commented on a change in pull request #3026:
URL: https://github.com/apache/hadoop/pull/3026#discussion_r643843222
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java
##
@@ -326,4 +345,37 @@ private void incrementCounter(AbfsStatistic statistic,
long value) {
abfsCounters.incrementCounter(statistic, value);
}
}
+
+ private void updateRequestHeaders() {
Review comment:
AbfsRestOperation is not supposed to have any API specific
implementations or handling. Come up with a LeaseRetryPolicy that will be used
when executing operations that pass in leaseId. That can control retry count to
2 or how many ever reduced number of retries needed. Post that, if the request
still fails, retry policy should exit AbfsRestOperation and abfsClient should
take care of retrying from the respective method post updating request headers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 605107)
Time Spent: 2h 20m (was: 2h 10m)
> ABFS: Introduce Lease Operations with Append to provide single writer
> semantics
> ---
>
> Key: HADOOP-17590
> URL: https://issues.apache.org/jira/browse/HADOOP-17590
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Sneha Varma
>Assignee: Sneha Varma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> The lease operations will be introduced as part of Append, Flush to ensure
> the single writer semantics.
>
> Details:
> Acquire Lease will be introduced in Create, Auto-Renew, Acquire will be added
> to Append & Release, Auto-Renew, Acquire in Flush.
>
> Duration the creation of the file the lease will be acquired, as part of
> appends the lease will be auto-renewed & the lease can be released as part of
> flush.
>
> By default the lease duration will be of 60 seconds.
> "fs.azure.write.enforcelease" & "fs.azure.write.lease.duration" two configs
> will be introduced.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17590) ABFS: Introduce Lease Operations with Append to provide single writer semantics
[
https://issues.apache.org/jira/browse/HADOOP-17590?focusedWorklogId=605105&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605105
]
ASF GitHub Bot logged work on HADOOP-17590:
---
Author: ASF GitHub Bot
Created on: 02/Jun/21 10:49
Start Date: 02/Jun/21 10:49
Worklog Time Spent: 10m
Work Description: snvijaya commented on a change in pull request #3026:
URL: https://github.com/apache/hadoop/pull/3026#discussion_r643842683
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##
@@ -585,6 +589,26 @@ public AbfsRestOperation append(final String path, final
byte[] buffer,
}
}
+if (lease != null && lease.getLeaseID() != null &&
!lease.getLeaseID().isEmpty()) {
Review comment:
Add below member functions to AbfsLease :
1. boolean hasValidLease() - which will return value of ( lease.getLeaseID()
!= null && !lease.getLeaseID().isEmpty())
2. AddLeaseHeaders(List requestHeaders) - and move the
requestHeader conditional header add logic into it
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##
@@ -648,16 +672,33 @@ public boolean appendSuccessCheckOp(AbfsRestOperation op,
final String path,
}
public AbfsRestOperation flush(final String path, final long position,
boolean retainUncommittedData,
- boolean isClose, final String cachedSasToken,
final String leaseId)
+ boolean isClose, final String cachedSasToken,
AbfsLease lease)
throws AzureBlobFileSystemException {
final List requestHeaders = createDefaultHeaders();
addCustomerProvidedKeyHeaders(requestHeaders);
// JDK7 does not support PATCH, so to workaround the issue we will use
// PUT and specify the real method in the X-Http-Method-Override header.
requestHeaders.add(new AbfsHttpHeader(X_HTTP_METHOD_OVERRIDE,
HTTP_METHOD_PATCH));
-if (leaseId != null) {
- requestHeaders.add(new AbfsHttpHeader(X_MS_LEASE_ID, leaseId));
+
+if (lease != null && lease.getLeaseID() != null &&
!lease.getLeaseID().isEmpty()) {
Review comment:
Same as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 605105)
Time Spent: 2h 10m (was: 2h)
> ABFS: Introduce Lease Operations with Append to provide single writer
> semantics
> ---
>
> Key: HADOOP-17590
> URL: https://issues.apache.org/jira/browse/HADOOP-17590
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Sneha Varma
>Assignee: Sneha Varma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> The lease operations will be introduced as part of Append, Flush to ensure
> the single writer semantics.
>
> Details:
> Acquire Lease will be introduced in Create, Auto-Renew, Acquire will be added
> to Append & Release, Auto-Renew, Acquire in Flush.
>
> Duration the creation of the file the lease will be acquired, as part of
> appends the lease will be auto-renewed & the lease can be released as part of
> flush.
>
> By default the lease duration will be of 60 seconds.
> "fs.azure.write.enforcelease" & "fs.azure.write.lease.duration" two configs
> will be introduced.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17590) ABFS: Introduce Lease Operations with Append to provide single writer semantics
[ https://issues.apache.org/jira/browse/HADOOP-17590?focusedWorklogId=605103&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605103 ] ASF GitHub Bot logged work on HADOOP-17590: --- Author: ASF GitHub Bot Created on: 02/Jun/21 10:49 Start Date: 02/Jun/21 10:49 Worklog Time Spent: 10m Work Description: snvijaya commented on a change in pull request #3026: URL: https://github.com/apache/hadoop/pull/3026#discussion_r643842493 ## File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java ## @@ -561,17 +568,14 @@ public AbfsRestOperation renameIdempotencyCheckOp( } public AbfsRestOperation append(final String path, final byte[] buffer, - AppendRequestParameters reqParams, final String cachedSasToken) + AppendRequestParameters reqParams, final String cachedSasToken, AbfsLease lease) Review comment: Lease should be a member of AppendRequestParameters. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 605103) Time Spent: 2h (was: 1h 50m) > ABFS: Introduce Lease Operations with Append to provide single writer > semantics > --- > > Key: HADOOP-17590 > URL: https://issues.apache.org/jira/browse/HADOOP-17590 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sneha Varma >Assignee: Sneha Varma >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > The lease operations will be introduced as part of Append, Flush to ensure > the single writer semantics. > > Details: > Acquire Lease will be introduced in Create, Auto-Renew, Acquire will be added > to Append & Release, Auto-Renew, Acquire in Flush. > > Duration the creation of the file the lease will be acquired, as part of > appends the lease will be auto-renewed & the lease can be released as part of > flush. > > By default the lease duration will be of 60 seconds. > "fs.azure.write.enforcelease" & "fs.azure.write.lease.duration" two configs > will be introduced. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #3026: HADOOP-17590 ABFS: Introduce Lease Operations with Append to provide single writer semantics
snvijaya commented on a change in pull request #3026:
URL: https://github.com/apache/hadoop/pull/3026#discussion_r643843222
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java
##
@@ -326,4 +345,37 @@ private void incrementCounter(AbfsStatistic statistic,
long value) {
abfsCounters.incrementCounter(statistic, value);
}
}
+
+ private void updateRequestHeaders() {
Review comment:
AbfsRestOperation is not supposed to have any API specific
implementations or handling. Come up with a LeaseRetryPolicy that will be used
when executing operations that pass in leaseId. That can control retry count to
2 or how many ever reduced number of retries needed. Post that, if the request
still fails, retry policy should exit AbfsRestOperation and abfsClient should
take care of retrying from the respective method post updating request headers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #3026: HADOOP-17590 ABFS: Introduce Lease Operations with Append to provide single writer semantics
snvijaya commented on a change in pull request #3026:
URL: https://github.com/apache/hadoop/pull/3026#discussion_r643842683
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##
@@ -585,6 +589,26 @@ public AbfsRestOperation append(final String path, final
byte[] buffer,
}
}
+if (lease != null && lease.getLeaseID() != null &&
!lease.getLeaseID().isEmpty()) {
Review comment:
Add below member functions to AbfsLease :
1. boolean hasValidLease() - which will return value of ( lease.getLeaseID()
!= null && !lease.getLeaseID().isEmpty())
2. AddLeaseHeaders(List requestHeaders) - and move the
requestHeader conditional header add logic into it
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##
@@ -648,16 +672,33 @@ public boolean appendSuccessCheckOp(AbfsRestOperation op,
final String path,
}
public AbfsRestOperation flush(final String path, final long position,
boolean retainUncommittedData,
- boolean isClose, final String cachedSasToken,
final String leaseId)
+ boolean isClose, final String cachedSasToken,
AbfsLease lease)
throws AzureBlobFileSystemException {
final List requestHeaders = createDefaultHeaders();
addCustomerProvidedKeyHeaders(requestHeaders);
// JDK7 does not support PATCH, so to workaround the issue we will use
// PUT and specify the real method in the X-Http-Method-Override header.
requestHeaders.add(new AbfsHttpHeader(X_HTTP_METHOD_OVERRIDE,
HTTP_METHOD_PATCH));
-if (leaseId != null) {
- requestHeaders.add(new AbfsHttpHeader(X_MS_LEASE_ID, leaseId));
+
+if (lease != null && lease.getLeaseID() != null &&
!lease.getLeaseID().isEmpty()) {
Review comment:
Same as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17590) ABFS: Introduce Lease Operations with Append to provide single writer semantics
[
https://issues.apache.org/jira/browse/HADOOP-17590?focusedWorklogId=605102&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605102
]
ASF GitHub Bot logged work on HADOOP-17590:
---
Author: ASF GitHub Bot
Created on: 02/Jun/21 10:48
Start Date: 02/Jun/21 10:48
Worklog Time Spent: 10m
Work Description: snvijaya commented on a change in pull request #3026:
URL: https://github.com/apache/hadoop/pull/3026#discussion_r643842160
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##
@@ -1698,14 +1703,29 @@ private void updateInfiniteLeaseDirs() {
this.azureInfiniteLeaseDirSet.remove("");
}
- private AbfsLease maybeCreateLease(String relativePath)
+ private AbfsLease maybeCreateFiniteLease(String relativePath, boolean
isNamespaceEnabled)
throws AzureBlobFileSystemException {
boolean enableInfiniteLease = isInfiniteLeaseKey(relativePath);
-if (!enableInfiniteLease) {
- return null;
+AbfsLease lease = null;
+if (!enableInfiniteLease && abfsConfiguration.isLeaseEnforced() &&
isNamespaceEnabled) {
+ lease = new AbfsLease(client, relativePath, false);
Review comment:
Are there scenarios where a finite lease created needs to explicitly
call on acquireLease API ? If that dependency isnt there, it would be better to
create 2 child classes to AbfsLease as AbfsAutoRefreshingLease and
AbfsFiniteLease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 605102)
Time Spent: 1h 50m (was: 1h 40m)
> ABFS: Introduce Lease Operations with Append to provide single writer
> semantics
> ---
>
> Key: HADOOP-17590
> URL: https://issues.apache.org/jira/browse/HADOOP-17590
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Sneha Varma
>Assignee: Sneha Varma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> The lease operations will be introduced as part of Append, Flush to ensure
> the single writer semantics.
>
> Details:
> Acquire Lease will be introduced in Create, Auto-Renew, Acquire will be added
> to Append & Release, Auto-Renew, Acquire in Flush.
>
> Duration the creation of the file the lease will be acquired, as part of
> appends the lease will be auto-renewed & the lease can be released as part of
> flush.
>
> By default the lease duration will be of 60 seconds.
> "fs.azure.write.enforcelease" & "fs.azure.write.lease.duration" two configs
> will be introduced.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #3026: HADOOP-17590 ABFS: Introduce Lease Operations with Append to provide single writer semantics
snvijaya commented on a change in pull request #3026: URL: https://github.com/apache/hadoop/pull/3026#discussion_r643842493 ## File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java ## @@ -561,17 +568,14 @@ public AbfsRestOperation renameIdempotencyCheckOp( } public AbfsRestOperation append(final String path, final byte[] buffer, - AppendRequestParameters reqParams, final String cachedSasToken) + AppendRequestParameters reqParams, final String cachedSasToken, AbfsLease lease) Review comment: Lease should be a member of AppendRequestParameters. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17590) ABFS: Introduce Lease Operations with Append to provide single writer semantics
[
https://issues.apache.org/jira/browse/HADOOP-17590?focusedWorklogId=605101&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605101
]
ASF GitHub Bot logged work on HADOOP-17590:
---
Author: ASF GitHub Bot
Created on: 02/Jun/21 10:48
Start Date: 02/Jun/21 10:48
Worklog Time Spent: 10m
Work Description: snvijaya commented on a change in pull request #3026:
URL: https://github.com/apache/hadoop/pull/3026#discussion_r643841953
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##
@@ -521,12 +523,14 @@ public OutputStream createFile(final Path path,
isNamespaceEnabled ? getOctalNotation(permission) : null,
isNamespaceEnabled ? getOctalNotation(umask) : null,
isAppendBlob,
-null);
+null,
+lease);
}
perfInfo.registerResult(op.getResult()).registerSuccess(true);
- AbfsLease lease = maybeCreateLease(relativePath);
-
+ if (lease == null) {
Review comment:
Acquiring lease along with create/append enables atomic fetch of lease
with the store operation and prevent any parallel writers. If configured for
infinite lease, the path creation will happen without bundled lease acquiry and
rely on later acquireLease API call in AbfsLease.
AbfsLease background acquireLease should be scheduled post lease is acquired
along with create API (bundled) ? Infinite lease flow purely on acquireLease
API should be the case only if fs.azure.write.enforceLease is off.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 605101)
Time Spent: 1h 40m (was: 1.5h)
> ABFS: Introduce Lease Operations with Append to provide single writer
> semantics
> ---
>
> Key: HADOOP-17590
> URL: https://issues.apache.org/jira/browse/HADOOP-17590
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Sneha Varma
>Assignee: Sneha Varma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The lease operations will be introduced as part of Append, Flush to ensure
> the single writer semantics.
>
> Details:
> Acquire Lease will be introduced in Create, Auto-Renew, Acquire will be added
> to Append & Release, Auto-Renew, Acquire in Flush.
>
> Duration the creation of the file the lease will be acquired, as part of
> appends the lease will be auto-renewed & the lease can be released as part of
> flush.
>
> By default the lease duration will be of 60 seconds.
> "fs.azure.write.enforcelease" & "fs.azure.write.lease.duration" two configs
> will be introduced.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #3026: HADOOP-17590 ABFS: Introduce Lease Operations with Append to provide single writer semantics
snvijaya commented on a change in pull request #3026:
URL: https://github.com/apache/hadoop/pull/3026#discussion_r643842160
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##
@@ -1698,14 +1703,29 @@ private void updateInfiniteLeaseDirs() {
this.azureInfiniteLeaseDirSet.remove("");
}
- private AbfsLease maybeCreateLease(String relativePath)
+ private AbfsLease maybeCreateFiniteLease(String relativePath, boolean
isNamespaceEnabled)
throws AzureBlobFileSystemException {
boolean enableInfiniteLease = isInfiniteLeaseKey(relativePath);
-if (!enableInfiniteLease) {
- return null;
+AbfsLease lease = null;
+if (!enableInfiniteLease && abfsConfiguration.isLeaseEnforced() &&
isNamespaceEnabled) {
+ lease = new AbfsLease(client, relativePath, false);
Review comment:
Are there scenarios where a finite lease created needs to explicitly
call on acquireLease API ? If that dependency isnt there, it would be better to
create 2 child classes to AbfsLease as AbfsAutoRefreshingLease and
AbfsFiniteLease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #3026: HADOOP-17590 ABFS: Introduce Lease Operations with Append to provide single writer semantics
snvijaya commented on a change in pull request #3026:
URL: https://github.com/apache/hadoop/pull/3026#discussion_r643841953
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##
@@ -521,12 +523,14 @@ public OutputStream createFile(final Path path,
isNamespaceEnabled ? getOctalNotation(permission) : null,
isNamespaceEnabled ? getOctalNotation(umask) : null,
isAppendBlob,
-null);
+null,
+lease);
}
perfInfo.registerResult(op.getResult()).registerSuccess(true);
- AbfsLease lease = maybeCreateLease(relativePath);
-
+ if (lease == null) {
Review comment:
Acquiring lease along with create/append enables atomic fetch of lease
with the store operation and prevent any parallel writers. If configured for
infinite lease, the path creation will happen without bundled lease acquiry and
rely on later acquireLease API call in AbfsLease.
AbfsLease background acquireLease should be scheduled post lease is acquired
along with create API (bundled) ? Infinite lease flow purely on acquireLease
API should be the case only if fs.azure.write.enforceLease is off.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17199) Backport HADOOP-13230 list/getFileStatus changes for preserved directory markers
[ https://issues.apache.org/jira/browse/HADOOP-17199?focusedWorklogId=605100&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605100 ] ASF GitHub Bot logged work on HADOOP-17199: --- Author: ASF GitHub Bot Created on: 02/Jun/21 10:47 Start Date: 02/Jun/21 10:47 Worklog Time Spent: 10m Work Description: hadoop-yetus removed a comment on pull request #2269: URL: https://github.com/apache/hadoop/pull/2269#issuecomment-685933544 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 11m 10s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 10 new or modified test files. | ||| _ branch-2.10 Compile Tests _ | | +0 :ok: | mvndep | 2m 30s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 17m 27s | branch-2.10 passed | | +1 :green_heart: | compile | 16m 41s | branch-2.10 passed | | +1 :green_heart: | checkstyle | 2m 29s | branch-2.10 passed | | +1 :green_heart: | mvnsite | 2m 48s | branch-2.10 passed | | +1 :green_heart: | javadoc | 1m 55s | branch-2.10 passed | | +0 :ok: | spotbugs | 1m 23s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 4m 26s | branch-2.10 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 46s | the patch passed | | +1 :green_heart: | compile | 15m 24s | the patch passed | | -1 :x: | javac | 15m 24s | root generated 1 new + 1349 unchanged - 1 fixed = 1350 total (was 1350) | | -0 :warning: | checkstyle | 2m 21s | root: The patch generated 53 new + 10 unchanged - 0 fixed = 63 total (was 10) | | +1 :green_heart: | mvnsite | 2m 17s | the patch passed | | -1 :x: | whitespace | 0m 0s | The patch has 10 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | javadoc | 2m 10s | the patch passed | | -1 :x: | findbugs | 0m 51s | hadoop-common in the patch failed. | | -1 :x: | findbugs | 0m 42s | hadoop-aws in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 0m 32s | hadoop-common in the patch failed. | | -1 :x: | unit | 0m 42s | hadoop-aws in the patch failed. | | +1 :green_heart: | asflicense | 0m 39s | The patch does not generate ASF License warnings. | | | | 93m 45s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2269/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2269 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint | | uname | Linux df2fa77f2ee9 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | branch-2.10 / ecf88b9 | | Default Java | Oracle Corporation-1.7.0_95-b00 | | javac | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2269/1/artifact/out/diff-compile-javac-root.txt | | checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2269/1/artifact/out/diff-checkstyle-root.txt | | whitespace | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2269/1/artifact/out/whitespace-eol.txt | | findbugs | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2269/1/artifact/out/patch-findbugs-hadoop-common-project_hadoop-common.txt | | findbugs | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2269/1/artifact/out/patch-findbugs-hadoop-tools_hadoop-aws.txt | | unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2269/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt | | unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2269/1/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2269/1/testReport/ | | Max. process+thread count | 92 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . | | Console outpu
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2269: HADOOP-17199. S3A Directory Marker HADOOP-13230 backport to branch-2
hadoop-yetus removed a comment on pull request #2269: URL: https://github.com/apache/hadoop/pull/2269#issuecomment-685933544 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 11m 10s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 10 new or modified test files. | ||| _ branch-2.10 Compile Tests _ | | +0 :ok: | mvndep | 2m 30s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 17m 27s | branch-2.10 passed | | +1 :green_heart: | compile | 16m 41s | branch-2.10 passed | | +1 :green_heart: | checkstyle | 2m 29s | branch-2.10 passed | | +1 :green_heart: | mvnsite | 2m 48s | branch-2.10 passed | | +1 :green_heart: | javadoc | 1m 55s | branch-2.10 passed | | +0 :ok: | spotbugs | 1m 23s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 4m 26s | branch-2.10 passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 1m 46s | the patch passed | | +1 :green_heart: | compile | 15m 24s | the patch passed | | -1 :x: | javac | 15m 24s | root generated 1 new + 1349 unchanged - 1 fixed = 1350 total (was 1350) | | -0 :warning: | checkstyle | 2m 21s | root: The patch generated 53 new + 10 unchanged - 0 fixed = 63 total (was 10) | | +1 :green_heart: | mvnsite | 2m 17s | the patch passed | | -1 :x: | whitespace | 0m 0s | The patch has 10 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 :green_heart: | javadoc | 2m 10s | the patch passed | | -1 :x: | findbugs | 0m 51s | hadoop-common in the patch failed. | | -1 :x: | findbugs | 0m 42s | hadoop-aws in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 0m 32s | hadoop-common in the patch failed. | | -1 :x: | unit | 0m 42s | hadoop-aws in the patch failed. | | +1 :green_heart: | asflicense | 0m 39s | The patch does not generate ASF License warnings. | | | | 93m 45s | | | Subsystem | Report/Notes | |--:|:-| | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2269/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/2269 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint | | uname | Linux df2fa77f2ee9 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | branch-2.10 / ecf88b9 | | Default Java | Oracle Corporation-1.7.0_95-b00 | | javac | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2269/1/artifact/out/diff-compile-javac-root.txt | | checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2269/1/artifact/out/diff-checkstyle-root.txt | | whitespace | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2269/1/artifact/out/whitespace-eol.txt | | findbugs | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2269/1/artifact/out/patch-findbugs-hadoop-common-project_hadoop-common.txt | | findbugs | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2269/1/artifact/out/patch-findbugs-hadoop-tools_hadoop-aws.txt | | unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2269/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt | | unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2269/1/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2269/1/testReport/ | | Max. process+thread count | 92 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2269/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.0.1 | | Powered by | Apache Yetus 0.12.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. Fo
[jira] [Work logged] (HADOOP-17590) ABFS: Introduce Lease Operations with Append to provide single writer semantics
[ https://issues.apache.org/jira/browse/HADOOP-17590?focusedWorklogId=605099&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605099 ] ASF GitHub Bot logged work on HADOOP-17590: --- Author: ASF GitHub Bot Created on: 02/Jun/21 10:46 Start Date: 02/Jun/21 10:46 Worklog Time Spent: 10m Work Description: snvijaya commented on a change in pull request #3026: URL: https://github.com/apache/hadoop/pull/3026#discussion_r643840786 ## File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java ## @@ -506,13 +506,15 @@ public OutputStream createFile(final Path path, triggerConditionalCreateOverwrite = true; } + AbfsLease lease = maybeCreateFiniteLease(relativePath, isNamespaceEnabled); Review comment: If the path qualifies for infinteLease as per config settings, this will still create a finite lease ? Call to a method createLease() that will first check for infiniteLease setting before defaulting to finite lease (if write lease config is enabled), would be the expectation ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 605099) Time Spent: 1.5h (was: 1h 20m) > ABFS: Introduce Lease Operations with Append to provide single writer > semantics > --- > > Key: HADOOP-17590 > URL: https://issues.apache.org/jira/browse/HADOOP-17590 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sneha Varma >Assignee: Sneha Varma >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > The lease operations will be introduced as part of Append, Flush to ensure > the single writer semantics. > > Details: > Acquire Lease will be introduced in Create, Auto-Renew, Acquire will be added > to Append & Release, Auto-Renew, Acquire in Flush. > > Duration the creation of the file the lease will be acquired, as part of > appends the lease will be auto-renewed & the lease can be released as part of > flush. > > By default the lease duration will be of 60 seconds. > "fs.azure.write.enforcelease" & "fs.azure.write.lease.duration" two configs > will be introduced. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17727) Modularize docker images
[ https://issues.apache.org/jira/browse/HADOOP-17727?focusedWorklogId=605098&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605098 ] ASF GitHub Bot logged work on HADOOP-17727: --- Author: ASF GitHub Bot Created on: 02/Jun/21 10:46 Start Date: 02/Jun/21 10:46 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on pull request #3043: URL: https://github.com/apache/hadoop/pull/3043#issuecomment-852923218 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3043/22/console in case of problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 605098) Time Spent: 8.5h (was: 8h 20m) > Modularize docker images > > > Key: HADOOP-17727 > URL: https://issues.apache.org/jira/browse/HADOOP-17727 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.4.0 >Reporter: Gautham Banasandra >Assignee: Gautham Banasandra >Priority: Major > Labels: pull-request-available > Time Spent: 8.5h > Remaining Estimate: 0h > > We're now creating the *Dockerfile*s for different platforms. We need a way > to manage the packages in a clean way as maintaining the packages for all the > different environments becomes cumbersome. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #3026: HADOOP-17590 ABFS: Introduce Lease Operations with Append to provide single writer semantics
snvijaya commented on a change in pull request #3026: URL: https://github.com/apache/hadoop/pull/3026#discussion_r643840786 ## File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java ## @@ -506,13 +506,15 @@ public OutputStream createFile(final Path path, triggerConditionalCreateOverwrite = true; } + AbfsLease lease = maybeCreateFiniteLease(relativePath, isNamespaceEnabled); Review comment: If the path qualifies for infinteLease as per config settings, this will still create a finite lease ? Call to a method createLease() that will first check for infiniteLease setting before defaulting to finite lease (if write lease config is enabled), would be the expectation ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3043: HADOOP-17727. Modularize docker images
hadoop-yetus commented on pull request #3043: URL: https://github.com/apache/hadoop/pull/3043#issuecomment-852923218 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3043/22/console in case of problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on pull request #3062: HDFS-16048. RBF: Print network topology on the router web
tomscut commented on pull request #3062: URL: https://github.com/apache/hadoop/pull/3062#issuecomment-852906104 Those failed unit tests work fine locally. https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3062/5/testReport/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3062: HDFS-16048. RBF: Print network topology on the router web
hadoop-yetus commented on pull request #3062:
URL: https://github.com/apache/hadoop/pull/3062#issuecomment-852894737
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 37s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 33s | | trunk passed |
| +1 :green_heart: | compile | 5m 16s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 4m 50s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 15s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 58s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 33s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 19s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 29s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 57s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 46s | | the patch passed |
| +1 :green_heart: | compile | 5m 10s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 5m 10s | | the patch passed |
| +1 :green_heart: | compile | 4m 40s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 4m 40s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 9s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 50s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 21s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 11s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 40s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 49s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 347m 27s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3062/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 24m 16s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 40s | | The patch does not
generate ASF License warnings. |
| | | 488m 52s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| |
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3062/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3062 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux a618afa0831e 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13
01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 05ca8ba36da36bae2e974bad43902681500ff2dc |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hado
[GitHub] [hadoop] hadoop-yetus commented on pull request #2894: HDFS-15966. Empty the statistical parameters when emptying the redundant queue
hadoop-yetus commented on pull request #2894:
URL: https://github.com/apache/hadoop/pull/2894#issuecomment-852879251
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 21m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 49s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 56s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 27s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 16s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 49s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 19s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 19s | | the patch passed |
| +1 :green_heart: | compile | 1m 8s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 47s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 18s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 49s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 351m 49s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2894/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 466m 18s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2894/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2894 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 92858aadc559 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9
06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8cfff7e1b0dcd5a7e1d79edabfcd9b7fd83575cf |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2894/4/testReport/ |
| Max. process+thread count | 2052 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop
[GitHub] [hadoop] wzhallright opened a new pull request #3066: HDFS-16053. Make the way of get heartbeat interval from conf consiste…
wzhallright opened a new pull request #3066: URL: https://github.com/apache/hadoop/pull/3066 Make the way of get heartbeat interval from conf consistent between Balancer and TestBalancer ## NOTICE JIRA: https://issues.apache.org/jira/browse/HDFS-16053 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17139) Re-enable optimized copyFromLocal implementation in S3AFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17355593#comment-17355593 ] Bogdan Stolojan commented on HADOOP-17139: -- Would like to have a look at this one if that's ok > Re-enable optimized copyFromLocal implementation in S3AFileSystem > - > > Key: HADOOP-17139 > URL: https://issues.apache.org/jira/browse/HADOOP-17139 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0, 3.2.1 >Reporter: Sahil Takiar >Priority: Minor > > It looks like HADOOP-15932 disabled the optimized copyFromLocal > implementation in S3A for correctness reasons. innerCopyFromLocalFile should > be fixed and re-enabled. The current implementation uses > FileSystem.copyFromLocal which will open an input stream from the local fs > and an output stream to the destination fs, and then call IOUtils.copyBytes. > With default configs, this will cause S3A to read the file into memory, write > it back to a file on the local fs, and then when the file is closed, upload > it to S3. > The optimized version of copyFromLocal in innerCopyFromLocalFile, directly > creates a PutObjectRequest request with the local file as the input. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
