[GitHub] [hadoop] umamaheswararao commented on pull request #2058: HDFS-15394. Add all available fs.viewfs.overload.scheme.target..impl classes in core-default.xml bydefault.

2020-06-05 Thread GitBox
umamaheswararao commented on pull request #2058: URL: https://github.com/apache/hadoop/pull/2058#issuecomment-639987530 I see this test is asserting xml vs config class. However, the idea is not to expose this configurations to users. I am building the key dynamically in using final

[GitHub] [hadoop] umamaheswararao merged pull request #2057: HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme.

2020-06-05 Thread GitBox
umamaheswararao merged pull request #2057: URL: https://github.com/apache/hadoop/pull/2057 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [hadoop] ayushtkn commented on pull request #2058: HDFS-15394. Add all available fs.viewfs.overload.scheme.target..impl classes in core-default.xml bydefault.

2020-06-05 Thread GitBox
ayushtkn commented on pull request #2058: URL: https://github.com/apache/hadoop/pull/2058#issuecomment-639986361 Thanx @umamaheswararao for the work here. * The TestCommonConfigurationFields is related, It expects the configurations to be added in CommonConfigurationKeys.java as well

[GitHub] [hadoop] ayushtkn commented on pull request #2057: HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme.

2020-06-05 Thread GitBox
ayushtkn commented on pull request #2057: URL: https://github.com/apache/hadoop/pull/2057#issuecomment-639982587 Thanx, the test failures aren't related This is an automated message from the Apache Git Service. To respond to

[GitHub] [hadoop] hadoop-yetus commented on pull request #2058: HDFS-15394. Add all available fs.viewfs.overload.scheme.target..impl classes in core-default.xml bydefault.

2020-06-05 Thread GitBox
hadoop-yetus commented on pull request #2058: URL: https://github.com/apache/hadoop/pull/2058#issuecomment-639980802 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m

[GitHub] [hadoop] hadoop-yetus commented on pull request #2058: HDFS-15394. Add all available fs.viewfs.overload.scheme.target..impl classes in core-default.xml bydefault.

2020-06-05 Thread GitBox
hadoop-yetus commented on pull request #2058: URL: https://github.com/apache/hadoop/pull/2058#issuecomment-639977014 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m

[GitHub] [hadoop] umamaheswararao opened a new pull request #2058: HDFS-15394. Add all available fs.viewfs.overload.scheme.target..impl classes in core-default.xml bydefault.

2020-06-05 Thread GitBox
umamaheswararao opened a new pull request #2058: URL: https://github.com/apache/hadoop/pull/2058 https://issues.apache.org/jira/browse/HDFS-15394 This is an automated message from the Apache Git Service. To respond to the

[GitHub] [hadoop] umamaheswararao commented on pull request #2057: HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme.

2020-06-05 Thread GitBox
umamaheswararao commented on pull request #2057: URL: https://github.com/apache/hadoop/pull/2057#issuecomment-639951401 Most of these tests failed with the OOM errors. Could you please confirm the failed tests are running fine? Thank you

[GitHub] [hadoop] umamaheswararao edited a comment on pull request #2057: HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme.

2020-06-05 Thread GitBox
umamaheswararao edited a comment on pull request #2057: URL: https://github.com/apache/hadoop/pull/2057#issuecomment-639776462 @ayushtkn, you are right. I am +1 on this changes. Since -setBalancerBandwidth is a sub option of dfsadmin and this was already covered in doc generically. So,

[jira] [Resolved] (HADOOP-17029) ViewFS does not return correct user/group and ACL

2020-06-05 Thread Uma Maheswara Rao G (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G resolved HADOOP-17029. -- Hadoop Flags: Reviewed Resolution: Fixed Thank you [~abhishekd] for

[GitHub] [hadoop] umamaheswararao merged pull request #2019: HADOOP-17029. Return correct permission and owner for listing on internal directories in ViewFs

2020-06-05 Thread GitBox
umamaheswararao merged pull request #2019: URL: https://github.com/apache/hadoop/pull/2019 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [hadoop] hadoop-yetus commented on pull request #2057: HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme.

2020-06-05 Thread GitBox
hadoop-yetus commented on pull request #2057: URL: https://github.com/apache/hadoop/pull/2057#issuecomment-639844273 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m

[GitHub] [hadoop] hadoop-yetus commented on pull request #2054: HDFS-15386 ReplicaNotFoundException keeps happening in DN after remov…

2020-06-05 Thread GitBox
hadoop-yetus commented on pull request #2054: URL: https://github.com/apache/hadoop/pull/2054#issuecomment-639822333 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 22m

[GitHub] [hadoop] umamaheswararao edited a comment on pull request #2057: HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme.

2020-06-05 Thread GitBox
umamaheswararao edited a comment on pull request #2057: URL: https://github.com/apache/hadoop/pull/2057#issuecomment-639776462 @ayushtkn, you are right. I am +1 on this changes. Since -setBalancerBandwidth is a subcommand of dfsadmin and this was already covered in doc. So, I will take

[GitHub] [hadoop] umamaheswararao commented on pull request #2057: HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme.

2020-06-05 Thread GitBox
umamaheswararao commented on pull request #2057: URL: https://github.com/apache/hadoop/pull/2057#issuecomment-639776462 @ayushtkn, you are right. -setBalancerBancdwidthNeed need not be a separate sub heading. You can just add note in dfsadmin itself. Later I will add remaining

[GitHub] [hadoop] abhishekdas99 commented on pull request #2019: HADOOP-17029. Return correct permission and owner for listing on internal directories in ViewFs

2020-06-05 Thread GitBox
abhishekdas99 commented on pull request #2019: URL: https://github.com/apache/hadoop/pull/2019#issuecomment-639749453 > +1 Looks good to me. Thanks for working on it. If no other objections, I will commit it in some time. Thanks @umamaheswararao. Sorry for the delay

[GitHub] [hadoop] ayushtkn commented on pull request #2057: HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme.

2020-06-05 Thread GitBox
ayushtkn commented on pull request #2057: URL: https://github.com/apache/hadoop/pull/2057#issuecomment-639732976 Thanx @umamaheswararao will update the document, just a minor doubt on this. got little confused bq. Or we can keep a heading "HDFS commands with ViewFSOverloadScheme" then

[GitHub] [hadoop] umamaheswararao commented on pull request #2019: HADOOP-17029. Return correct permission and owner for listing on internal directories in ViewFs

2020-06-05 Thread GitBox
umamaheswararao commented on pull request #2019: URL: https://github.com/apache/hadoop/pull/2019#issuecomment-639723041 Looks good to me. Thanks for working on it. If no other objections, I will commit it in some time. This

[GitHub] [hadoop] umamaheswararao edited a comment on pull request #2019: HADOOP-17029. Return correct permission and owner for listing on internal directories in ViewFs

2020-06-05 Thread GitBox
umamaheswararao edited a comment on pull request #2019: URL: https://github.com/apache/hadoop/pull/2019#issuecomment-639723041 +1 Looks good to me. Thanks for working on it. If no other objections, I will commit it in some time.

[GitHub] [hadoop] umamaheswararao commented on pull request #2057: HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme.

2020-06-05 Thread GitBox
umamaheswararao commented on pull request #2057: URL: https://github.com/apache/hadoop/pull/2057#issuecomment-639708773 Thanks @ayushtkn for reporting it. LGTM. However I have one point, can we add a note about balancer bandwidth command with overloadScheme when they need to pass -fs?

[GitHub] [hadoop] ayushtkn opened a new pull request #2057: HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme.

2020-06-05 Thread GitBox
ayushtkn opened a new pull request #2057: URL: https://github.com/apache/hadoop/pull/2057 https://issues.apache.org/jira/browse/HDFS-15389 This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [hadoop] umamaheswararao merged pull request #2053: HDFS-15330. Document the ViewFSOverloadScheme details in ViewFS guide

2020-06-05 Thread GitBox
umamaheswararao merged pull request #2053: URL: https://github.com/apache/hadoop/pull/2053 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [hadoop] hadoop-yetus commented on pull request #2053: HDFS-15330. Document the ViewFSOverloadScheme details in ViewFS guide

2020-06-05 Thread GitBox
hadoop-yetus commented on pull request #2053: URL: https://github.com/apache/hadoop/pull/2053#issuecomment-639661187 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m

[jira] [Comment Edited] (HADOOP-17066) S3A staging committer committing duplicate files

2020-06-05 Thread Brandon (Jira)
option is directory 20/06/05 07:24:39 DEBUG AbstractS3ACommitter: Task committer attempt_20200605072439__m_00_0 instantiated for job "(anonymous)" ID job_20200605072439_ with destination s3://bucket/spark/tables/example/table-20200605-26635-TbaguBgC8ef/other_namespace/tes

[GitHub] [hadoop] hadoop-yetus commented on pull request #2056: HADOOP-17065. Adding Network Counters in ABFS

2020-06-05 Thread GitBox
hadoop-yetus commented on pull request #2056: URL: https://github.com/apache/hadoop/pull/2056#issuecomment-639640819 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m

[GitHub] [hadoop] ayushtkn commented on pull request #2053: HDFS-15330. Document the ViewFSOverloadScheme details in ViewFS guide

2020-06-05 Thread GitBox
ayushtkn commented on pull request #2053: URL: https://github.com/apache/hadoop/pull/2053#issuecomment-639635146 Thanx @umamaheswararao for the update. +1 This is an automated message from the Apache Git Service. To

[GitHub] [hadoop] umamaheswararao commented on pull request #2053: HDFS-15330. Document the ViewFSOverloadScheme details in ViewFS guide

2020-06-05 Thread GitBox
umamaheswararao commented on pull request #2053: URL: https://github.com/apache/hadoop/pull/2053#issuecomment-639620945 Thanks @ayushtkn for review! I have updated with typos fixed. Please take a look at it. This is an

[jira] [Commented] (HADOOP-17066) S3A staging committer committing duplicate files

2020-06-05 Thread Brandon (Jira)
AbstractS3ACommitter: Task committer attempt_20200605072439__m_00_0 instantiated for job "(anonymous)" ID job_20200605072439_ with destination s3://bucket/spark/tables/planex/table-20200605-26635-TbaguBgC8ef/other_namespace/test_a.orc 20/06/05 07:24:39 DEBUG AbstractS3

[jira] [Commented] (HADOOP-17066) S3A staging committer committing duplicate files

2020-06-05 Thread Brandon (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126931#comment-17126931 ] Brandon commented on HADOOP-17066: -- Attaching debug logs. This is from a simple Spark application that

[GitHub] [hadoop] hadoop-yetus commented on pull request #1679: HDFS-13934. Multipart uploaders to be created through FileSystem/FileContext.

2020-06-05 Thread GitBox
hadoop-yetus commented on pull request #1679: URL: https://github.com/apache/hadoop/pull/1679#issuecomment-639598730 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m

[jira] [Commented] (HADOOP-9851) dfs -chown does not like "+" plus sign in user name

2020-06-05 Thread Ayush Saxena (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-9851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126903#comment-17126903 ] Ayush Saxena commented on HADOOP-9851: -- Thanx [~boky01] for the patch. Well I think adding support

[jira] [Issue Comment Deleted] (HADOOP-13867) FilterFileSystem should override rename(.., options) to take effect of Rename options called via FilterFileSystem implementations

2020-06-05 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-13867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HADOOP-13867: Comment: was deleted (was: I just found out this breaks {{LocalFileSystem}}. If the three

[GitHub] [hadoop] hadoop-yetus commented on pull request #2055: HDFS-15393: Review of PendingReconstructionBlocks

2020-06-05 Thread GitBox
hadoop-yetus commented on pull request #2055: URL: https://github.com/apache/hadoop/pull/2055#issuecomment-639582280 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 26m

[GitHub] [hadoop] belugabehr opened a new pull request #2055: HDFS-15393: Review of PendingReconstructionBlocks

2020-06-05 Thread GitBox
belugabehr opened a new pull request #2055: URL: https://github.com/apache/hadoop/pull/2055 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [hadoop] hadoop-yetus commented on pull request #2054: HDFS-15386 ReplicaNotFoundException keeps happening in DN after remov…

2020-06-05 Thread GitBox
hadoop-yetus commented on pull request #2054: URL: https://github.com/apache/hadoop/pull/2054#issuecomment-639487011 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m

[GitHub] [hadoop] brfrn169 opened a new pull request #2054: HDFS-15386 ReplicaNotFoundException keeps happening in DN after remov…

2020-06-05 Thread GitBox
brfrn169 opened a new pull request #2054: URL: https://github.com/apache/hadoop/pull/2054 …ing multiple DN's data directories This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [hadoop] hadoop-yetus commented on pull request #2019: HADOOP-17029. Return correct permission and owner for listing on internal directories in ViewFs

2020-06-05 Thread GitBox
hadoop-yetus commented on pull request #2019: URL: https://github.com/apache/hadoop/pull/2019#issuecomment-639475887 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m

[jira] [Commented] (HADOOP-14698) Make copyFromLocal's -t option available for put as well

2020-06-05 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-14698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126716#comment-17126716 ] Stephen O'Donnell commented on HADOOP-14698: Now committed to branch-3.0 too. > Make

[jira] [Updated] (HADOOP-14698) Make copyFromLocal's -t option available for put as well

2020-06-05 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-14698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell updated HADOOP-14698: --- Fix Version/s: 3.0.4 > Make copyFromLocal's -t option available for put as well >

[GitHub] [hadoop] ayushtkn commented on a change in pull request #2053: HDFS-15330. Document the ViewFSOverloadScheme details in ViewFS guide

2020-06-05 Thread GitBox
ayushtkn commented on a change in pull request #2053: URL: https://github.com/apache/hadoop/pull/2053#discussion_r435860568 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md ## @@ -693,4 +693,42 @@ Usage: `hdfs debug recoverLease -path [-retries

[GitHub] [hadoop] abhishekdas99 commented on a change in pull request #2019: HADOOP-17029. Return correct permission and owner for listing on internal directories in ViewFs

2020-06-05 Thread GitBox
abhishekdas99 commented on a change in pull request #2019: URL: https://github.com/apache/hadoop/pull/2019#discussion_r435856451 ## File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java ## @@ -1200,13 +1200,24 @@ public

[GitHub] [hadoop] sodonnel merged pull request #2052: HDFS-15386. ReplicaNotFoundException keeps happening in DN after removing multiple DN's data directories.

2020-06-05 Thread GitBox
sodonnel merged pull request #2052: URL: https://github.com/apache/hadoop/pull/2052 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[jira] [Created] (HADOOP-17067) AuthenticatedURL.Token does not store the token if there are redirects.

2020-06-05 Thread Harish JP (Jira)
Harish JP created HADOOP-17067: -- Summary: AuthenticatedURL.Token does not store the token if there are redirects. Key: HADOOP-17067 URL: https://issues.apache.org/jira/browse/HADOOP-17067 Project:

[jira] [Created] (HADOOP-17066) S3A staging committer committing duplicate files

2020-06-05 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-17066: --- Summary: S3A staging committer committing duplicate files Key: HADOOP-17066 URL: https://issues.apache.org/jira/browse/HADOOP-17066 Project: Hadoop Common

[jira] [Commented] (HADOOP-17050) Add support for multiple delegation tokens in S3AFilesystem

2020-06-05 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126549#comment-17126549 ] Gabor Bota commented on HADOOP-17050: - so it will be the "in s3a" option from above > Add support