[jira] [Commented] (HADOOP-18705) hadoop-azure: AzureBlobFileSystem should exclude incompatible credential providers when binding DelegationTokenManagers

2023-04-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713901#comment-17713901 ] ASF GitHub Bot commented on HADOOP-18705: - tomicooler commented on PR #5560: URL:

[GitHub] [hadoop] tomicooler commented on pull request #5560: HADOOP-18705. hadoop-azure: AzureBlobFileSystem should exclude incomp…

2023-04-18 Thread via GitHub
tomicooler commented on PR #5560: URL: https://github.com/apache/hadoop/pull/5560#issuecomment-1514156973 @steveloughran Thanks for the review. I just read the testing_azure.md, I haven't run the integration tests yet. -- This is an automated message from the Apache Git Service. To

[GitHub] [hadoop] tomicooler commented on a diff in pull request #5560: HADOOP-18705. hadoop-azure: AzureBlobFileSystem should exclude incomp…

2023-04-18 Thread via GitHub
tomicooler commented on code in PR #5560: URL: https://github.com/apache/hadoop/pull/5560#discussion_r1170825511 ## hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java: ## @@ -196,6 +196,11 @@ public void initialize(URI uri,

[jira] [Commented] (HADOOP-18705) hadoop-azure: AzureBlobFileSystem should exclude incompatible credential providers when binding DelegationTokenManagers

2023-04-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713900#comment-17713900 ] ASF GitHub Bot commented on HADOOP-18705: - tomicooler commented on code in PR #5560: URL:

[jira] [Commented] (HADOOP-18705) hadoop-azure: AzureBlobFileSystem should exclude incompatible credential providers when binding DelegationTokenManagers

2023-04-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713897#comment-17713897 ] ASF GitHub Bot commented on HADOOP-18705: - tomicooler commented on code in PR #5560: URL:

[jira] [Commented] (HADOOP-18705) hadoop-azure: AzureBlobFileSystem should exclude incompatible credential providers when binding DelegationTokenManagers

2023-04-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713898#comment-17713898 ] ASF GitHub Bot commented on HADOOP-18705: - tomicooler commented on code in PR #5560: URL:

[GitHub] [hadoop] tomicooler commented on a diff in pull request #5560: HADOOP-18705. hadoop-azure: AzureBlobFileSystem should exclude incomp…

2023-04-18 Thread via GitHub
tomicooler commented on code in PR #5560: URL: https://github.com/apache/hadoop/pull/5560#discussion_r1170822736 ## hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemConfiguration.java: ## @@ -0,0 +1,42 @@ +/* + * Licensed to the

[GitHub] [hadoop] tomicooler commented on a diff in pull request #5560: HADOOP-18705. hadoop-azure: AzureBlobFileSystem should exclude incomp…

2023-04-18 Thread via GitHub
tomicooler commented on code in PR #5560: URL: https://github.com/apache/hadoop/pull/5560#discussion_r1170822606 ## hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemConfiguration.java: ## @@ -0,0 +1,42 @@ +/* + * Licensed to the

[GitHub] [hadoop] hadoop-yetus commented on pull request #5568: HDFS-16653.Add error message for maxEvictableMmapedSize related Precondition check suite.

2023-04-18 Thread via GitHub
hadoop-yetus commented on PR #5568: URL: https://github.com/apache/hadoop/pull/5568#issuecomment-1514108239 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5551: YARN-11378. [Federation] Support checkForDecommissioningNodes、refreshClusterMaxPriority API's for Federation.

2023-04-18 Thread via GitHub
hadoop-yetus commented on PR #5551: URL: https://github.com/apache/hadoop/pull/5551#issuecomment-1514095343 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] LiuGuH commented on a diff in pull request #5552: HDFS-16979. RBF: Add dfsrouter port in hdfsauditlog

2023-04-18 Thread via GitHub
LiuGuH commented on code in PR #5552: URL: https://github.com/apache/hadoop/pull/5552#discussion_r1170763729 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java: ## @@ -453,15 +453,15 @@ private void logAuditEvent(boolean

[GitHub] [hadoop] Likkey opened a new pull request, #5569: HDFS-16697.Add code to check for minimumRedundantVolumes.

2023-04-18 Thread via GitHub
Likkey opened a new pull request, #5569: URL: https://github.com/apache/hadoop/pull/5569 ### Description of PR It is found that “dfs.namenode.resource.checked.volumes.minimum” lacks a condition check and an associated exception handling mechanism, which makes it impossible to find

[GitHub] [hadoop] Likkey opened a new pull request, #5568: HDFS-16653.Add error message for maxEvictableMmapedSize related Precondition check suite.

2023-04-18 Thread via GitHub
Likkey opened a new pull request, #5568: URL: https://github.com/apache/hadoop/pull/5568 ### Description of PR When the configuration item “dfs.client.mmap.cache.size” is set to a negative number, it will cause /hadoop/bin hdfs dfsadmin -safemode provides all the operation options

[GitHub] [hadoop] Likkey closed pull request #4849: HDFS-16697.Add code to check for minimumRedundantVolumes.

2023-04-18 Thread via GitHub
Likkey closed pull request #4849: HDFS-16697.Add code to check for minimumRedundantVolumes. URL: https://github.com/apache/hadoop/pull/4849 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [hadoop] goiri commented on a diff in pull request #5552: HDFS-16979. RBF: Add dfsrouter port in hdfsauditlog

2023-04-18 Thread via GitHub
goiri commented on code in PR #5552: URL: https://github.com/apache/hadoop/pull/5552#discussion_r1170739964 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java: ## @@ -453,15 +453,15 @@ private void logAuditEvent(boolean

[GitHub] [hadoop] Likkey closed pull request #4848: HDFS-16653.Add error message for maxEvictableMmapedSize related Precondition check suite.

2023-04-18 Thread via GitHub
Likkey closed pull request #4848: HDFS-16653.Add error message for maxEvictableMmapedSize related Precondition check suite. URL: https://github.com/apache/hadoop/pull/4848 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [hadoop] goiri commented on a diff in pull request #5552: HDFS-16979. RBF: Add dfsrouter port in hdfsauditlog

2023-04-18 Thread via GitHub
goiri commented on code in PR #5552: URL: https://github.com/apache/hadoop/pull/5552#discussion_r1170739316 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java: ## @@ -453,15 +453,15 @@ private void logAuditEvent(boolean

[GitHub] [hadoop] Likkey closed pull request #4847: HDFS-16721.Improve the check code of “dfs.client.socket-timeout”.

2023-04-18 Thread via GitHub
Likkey closed pull request #4847: HDFS-16721.Improve the check code of “dfs.client.socket-timeout”. URL: https://github.com/apache/hadoop/pull/4847 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [hadoop] hadoop-yetus commented on pull request #5519: MAPREDUCE-7435. Manifest Committer OOM on abfs

2023-04-18 Thread via GitHub
hadoop-yetus commented on PR #5519: URL: https://github.com/apache/hadoop/pull/5519#issuecomment-1514011109 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5566: Bump jetty-server from 9.4.48.v20220622 to 10.0.14 in /hadoop-project

2023-04-18 Thread via GitHub
hadoop-yetus commented on PR #5566: URL: https://github.com/apache/hadoop/pull/5566#issuecomment-1513927807 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18701) Generic Build Improvements

2023-04-18 Thread Ayush Saxena (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713799#comment-17713799 ] Ayush Saxena commented on HADOOP-18701: --- yeps, should add it in hadoop-main. Some confusion:

[jira] [Updated] (HADOOP-18701) Generic Build Improvements

2023-04-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-18701: Labels: pull-request-available (was: ) > Generic Build Improvements >

[jira] [Commented] (HADOOP-18701) Generic Build Improvements

2023-04-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713798#comment-17713798 ] ASF GitHub Bot commented on HADOOP-18701: - ayushtkn opened a new pull request, #5567: URL:

[GitHub] [hadoop] ayushtkn opened a new pull request, #5567: HADOOP-18701. Generic Build Improvements.

2023-04-18 Thread via GitHub
ayushtkn opened a new pull request, #5567: URL: https://github.com/apache/hadoop/pull/5567 ### Description of PR **WIP:** Attempting basic improvements ### How was this patch tested? **WIP** Tried failIfNoSpecifiedTests prop by now ### For code changes:

[GitHub] [hadoop] slfan1989 commented on pull request #4963: YARN-11326. [Federation] Add RM FederationStateStoreService Metrics.

2023-04-18 Thread via GitHub
slfan1989 commented on PR #4963: URL: https://github.com/apache/hadoop/pull/4963#issuecomment-1513893947 @goiri Thank you very much for helping to review the code! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [hadoop] slfan1989 commented on pull request #5537: YARN-11438. [Federation] ZookeeperFederationStateStore Support Version.

2023-04-18 Thread via GitHub
slfan1989 commented on PR #5537: URL: https://github.com/apache/hadoop/pull/5537#issuecomment-1513894068 @goiri Thank you very much for helping to review the code! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [hadoop] dependabot[bot] opened a new pull request, #5566: Bump jetty-server from 9.4.48.v20220622 to 10.0.14 in /hadoop-project

2023-04-18 Thread via GitHub
dependabot[bot] opened a new pull request, #5566: URL: https://github.com/apache/hadoop/pull/5566 Bumps [jetty-server](https://github.com/eclipse/jetty.project) from 9.4.48.v20220622 to 10.0.14. Release notes Sourced from

[jira] [Commented] (HADOOP-18701) Generic Build Improvements

2023-04-18 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713765#comment-17713765 ] Steve Loughran commented on HADOOP-18701: - not sure if that surefire jira is the one, as our

[jira] [Updated] (HADOOP-18704) Support a "permissive" mode for secure clusters to allow "simple" auth clients

2023-04-18 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18704: Component/s: security > Support a "permissive" mode for secure clusters to allow

[jira] [Updated] (HADOOP-18704) Support a "permissive" mode for secure clusters to allow "simple" auth clients

2023-04-18 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18704: Issue Type: New Feature (was: Improvement) > Support a "permissive" mode for secure

[jira] [Updated] (HADOOP-18704) Support a "permissive" mode for secure clusters to allow "simple" auth clients

2023-04-18 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18704: Priority: Major (was: Minor) > Support a "permissive" mode for secure clusters to allow

[jira] [Commented] (HADOOP-18704) Support a "permissive" mode for secure clusters to allow "simple" auth clients

2023-04-18 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713757#comment-17713757 ] Steve Loughran commented on HADOOP-18704: - this is going to be up to the yarn/hdfs team to

[jira] [Commented] (HADOOP-18149) The FSDownload verifyAndCopy method doesn't support S3

2023-04-18 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713756#comment-17713756 ] Steve Loughran commented on HADOOP-18149: - why closing as invalid? > The FSDownload

[jira] [Commented] (HADOOP-18706) The temporary files for disk-block buffer aren't unique enough to recover partial uploads. 

2023-04-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713754#comment-17713754 ] ASF GitHub Bot commented on HADOOP-18706: - hadoop-yetus commented on PR #5563: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5563: HADOOP-18706: Improve S3ABlockOutputStream recovery

2023-04-18 Thread via GitHub
hadoop-yetus commented on PR #5563: URL: https://github.com/apache/hadoop/pull/5563#issuecomment-1513746171 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5554: HDFS-16978. RBF: Admin command to support bulk add of mount points

2023-04-18 Thread via GitHub
hadoop-yetus commented on PR #5554: URL: https://github.com/apache/hadoop/pull/5554#issuecomment-1513708832 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18706) The temporary files for disk-block buffer aren't unique enough to recover partial uploads. 

2023-04-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713742#comment-17713742 ] ASF GitHub Bot commented on HADOOP-18706: - cbevard1 commented on PR #5563: URL:

[GitHub] [hadoop] cbevard1 commented on pull request #5563: HADOOP-18706: Improve S3ABlockOutputStream recovery

2023-04-18 Thread via GitHub
cbevard1 commented on PR #5563: URL: https://github.com/apache/hadoop/pull/5563#issuecomment-1513699708 @steveloughran thanks for your feedback. I've added the span ID to the file name as you suggested for better debugging. > If you really want upload to be recoverable then you need

[jira] [Commented] (HADOOP-18691) Add a CallerContext getter on the Schedulable interface

2023-04-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713710#comment-17713710 ] ASF GitHub Bot commented on HADOOP-18691: - smengcl commented on PR #5540: URL:

[GitHub] [hadoop] smengcl commented on pull request #5540: HADOOP-18691. Add a CallerContext getter on the Schedulable interface

2023-04-18 Thread via GitHub
smengcl commented on PR #5540: URL: https://github.com/apache/hadoop/pull/5540#issuecomment-1513603521 Thanks @xBis7 . Latest changes lgtm. Looks like all comments from @steveloughran are addressed. I will merge after a few days or when Steve approves this. -- This is an automated

[jira] [Commented] (HADOOP-18706) The temporary files for disk-block buffer aren't unique enough to recover partial uploads. 

2023-04-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713708#comment-17713708 ] ASF GitHub Bot commented on HADOOP-18706: - hadoop-yetus commented on PR #5563: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5563: HADOOP-18706: Improve S3ABlockOutputStream recovery

2023-04-18 Thread via GitHub
hadoop-yetus commented on PR #5563: URL: https://github.com/apache/hadoop/pull/5563#issuecomment-1513592514 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] goiri merged pull request #5556: HDFS-16982 Use the right Quantiles Array for Inverse Quantiles snapshot

2023-04-18 Thread via GitHub
goiri merged PR #5556: URL: https://github.com/apache/hadoop/pull/5556 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5565: YARN-11467. RM failover may fail when the nodes.exclude-path file does not exist

2023-04-18 Thread via GitHub
hadoop-yetus commented on PR #5565: URL: https://github.com/apache/hadoop/pull/5565#issuecomment-1513564362 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] rdingankar commented on pull request #5556: HDFS-16982 Use the right Quantiles Array for Inverse Quantiles snapshot

2023-04-18 Thread via GitHub
rdingankar commented on PR #5556: URL: https://github.com/apache/hadoop/pull/5556#issuecomment-1513527413 Thanks for the review @goiri Can you also help in merging the change. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[jira] [Commented] (HADOOP-18425) [ABFS]: RenameFilePath Source File Not Found (404) error in retry loop

2023-04-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713679#comment-17713679 ] ASF GitHub Bot commented on HADOOP-18425: - steveloughran closed pull request #5485:

[GitHub] [hadoop] steveloughran closed pull request #5485: HADOOP-18425. ABFS rename resilience through etags

2023-04-18 Thread via GitHub
steveloughran closed pull request #5485: HADOOP-18425. ABFS rename resilience through etags URL: https://github.com/apache/hadoop/pull/5485 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[jira] [Commented] (HADOOP-18695) S3A: reject multipart copy requests when disabled

2023-04-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713678#comment-17713678 ] ASF GitHub Bot commented on HADOOP-18695: - steveloughran commented on PR #5548: URL:

[GitHub] [hadoop] steveloughran commented on pull request #5548: HADOOP-18695. S3A: reject multipart copy requests when disabled

2023-04-18 Thread via GitHub
steveloughran commented on PR #5548: URL: https://github.com/apache/hadoop/pull/5548#issuecomment-1513517379 I think this is ready to go in. more reviews please! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [hadoop] goiri merged pull request #4963: YARN-11326. [Federation] Add RM FederationStateStoreService Metrics.

2023-04-18 Thread via GitHub
goiri merged PR #4963: URL: https://github.com/apache/hadoop/pull/4963 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hadoop] goiri commented on pull request #5551: YARN-11378. [Federation] Support checkForDecommissioningNodes、refreshClusterMaxPriority API's for Federation.

2023-04-18 Thread via GitHub
goiri commented on PR #5551: URL: https://github.com/apache/hadoop/pull/5551#issuecomment-1513443941 Let's fix the checkstyle too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [hadoop] goiri commented on a diff in pull request #5554: HDFS-16978. RBF: Admin command to support bulk add of mount points

2023-04-18 Thread via GitHub
goiri commented on code in PR #5554: URL: https://github.com/apache/hadoop/pull/5554#discussion_r1170271873 ## hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java: ## @@ -462,6 +481,136 @@ public int run(String[] argv)

[jira] [Updated] (HADOOP-18399) S3A Prefetch - SingleFilePerBlockCache to use LocalDirAllocator

2023-04-18 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18399: Fix Version/s: 3.4.0 Resolution: Fixed Status: Resolved (was: Patch

[jira] [Updated] (HADOOP-18399) S3A Prefetch - SingleFilePerBlockCache to use LocalDirAllocator

2023-04-18 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18399: Summary: S3A Prefetch - SingleFilePerBlockCache to use LocalDirAllocator (was:

[GitHub] [hadoop] goiri merged pull request #5537: YARN-11438. [Federation] ZookeeperFederationStateStore Support Version.

2023-04-18 Thread via GitHub
goiri merged PR #5537: URL: https://github.com/apache/hadoop/pull/5537 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Commented] (HADOOP-18399) SingleFilePerBlockCache to use LocalDirAllocator for file allocation

2023-04-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713632#comment-17713632 ] ASF GitHub Bot commented on HADOOP-18399: - steveloughran merged PR #5054: URL:

[GitHub] [hadoop] steveloughran merged pull request #5054: HADOOP-18399. S3A Prefetch - SingleFilePerBlockCache to use LocalDirAllocator

2023-04-18 Thread via GitHub
steveloughran merged PR #5054: URL: https://github.com/apache/hadoop/pull/5054 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Commented] (HADOOP-18671) Add recoverLease(), setSafeMode(), isFileClosed() APIs to FileSystem

2023-04-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713631#comment-17713631 ] ASF GitHub Bot commented on HADOOP-18671: - steveloughran commented on PR #5553: URL:

[GitHub] [hadoop] steveloughran commented on pull request #5553: HADOOP-18671 Add recoverLease(), setSafeMode(), isFileClosed() APIs to FileSystem

2023-04-18 Thread via GitHub
steveloughran commented on PR #5553: URL: https://github.com/apache/hadoop/pull/5553#issuecomment-1513370122 there must still be a CommonPathCapabilities name as we will also need to have to consider about having * filterfs implement and pass through to the wrapped fs * viewfs to

[jira] [Commented] (HADOOP-18671) Add recoverLease(), setSafeMode(), isFileClosed() APIs to FileSystem

2023-04-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713628#comment-17713628 ] ASF GitHub Bot commented on HADOOP-18671: - steveloughran commented on code in PR #5553: URL:

[GitHub] [hadoop] steveloughran commented on a diff in pull request #5553: HADOOP-18671 Add recoverLease(), setSafeMode(), isFileClosed() APIs to FileSystem

2023-04-18 Thread via GitHub
steveloughran commented on code in PR #5553: URL: https://github.com/apache/hadoop/pull/5553#discussion_r1170210411 ## hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFileSystemContract.java: ## @@ -137,4 +142,30 @@ public void testRenameNonExistentPath()

[jira] [Commented] (HADOOP-18705) hadoop-azure: AzureBlobFileSystem should exclude incompatible credential providers when binding DelegationTokenManagers

2023-04-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713626#comment-17713626 ] ASF GitHub Bot commented on HADOOP-18705: - steveloughran commented on code in PR #5560: URL:

[GitHub] [hadoop] steveloughran commented on a diff in pull request #5560: HADOOP-18705. hadoop-azure: AzureBlobFileSystem should exclude incomp…

2023-04-18 Thread via GitHub
steveloughran commented on code in PR #5560: URL: https://github.com/apache/hadoop/pull/5560#discussion_r1169748503 ## hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemConfiguration.java: ## @@ -0,0 +1,42 @@ +/* + * Licensed to the

[jira] [Commented] (HADOOP-18706) The temporary files for disk-block buffer aren't unique enough to recover partial uploads. 

2023-04-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713618#comment-17713618 ] ASF GitHub Bot commented on HADOOP-18706: - steveloughran commented on code in PR #5563: URL:

[GitHub] [hadoop] steveloughran commented on a diff in pull request #5563: HADOOP-18706: Improve S3ABlockOutputStream recovery

2023-04-18 Thread via GitHub
steveloughran commented on code in PR #5563: URL: https://github.com/apache/hadoop/pull/5563#discussion_r1170129401 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ABlockOutputStream.java: ## @@ -726,7 +726,7 @@ public void hsync() throws IOException {

[GitHub] [hadoop] steveloughran commented on a diff in pull request #5563: HADOOP-18706: Improve S3ABlockOutputStream recovery

2023-04-18 Thread via GitHub
steveloughran commented on code in PR #5563: URL: https://github.com/apache/hadoop/pull/5563#discussion_r1170131112 ## hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestDataBlocks.java: ## @@ -51,7 +51,7 @@ public void testByteBufferIO() throws Throwable {

[jira] [Commented] (HADOOP-18706) The temporary files for disk-block buffer aren't unique enough to recover partial uploads. 

2023-04-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713617#comment-17713617 ] ASF GitHub Bot commented on HADOOP-18706: - steveloughran commented on code in PR #5563: URL:

[jira] [Commented] (HADOOP-18706) The temporary files for disk-block buffer aren't unique enough to recover partial uploads. 

2023-04-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713616#comment-17713616 ] ASF GitHub Bot commented on HADOOP-18706: - hadoop-yetus commented on PR #5563: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5563: HADOOP-18706: Improve S3ABlockOutputStream recovery

2023-04-18 Thread via GitHub
hadoop-yetus commented on PR #5563: URL: https://github.com/apache/hadoop/pull/5563#issuecomment-1513283429 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18694) Client.Connection#updateAddress needs to ensure that address is resolved before updating

2023-04-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713614#comment-17713614 ] ASF GitHub Bot commented on HADOOP-18694: - hadoop-yetus commented on PR #5542: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5542: HADOOP-18694. Client.Connection#updateAddress needs to ensure that address is resolved before updating

2023-04-18 Thread via GitHub
hadoop-yetus commented on PR #5542: URL: https://github.com/apache/hadoop/pull/5542#issuecomment-1513282177 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] cxzl25 opened a new pull request, #5565: YARN-11467. RM failover may fail when the nodes.exclude-path file does not exist

2023-04-18 Thread via GitHub
cxzl25 opened a new pull request, #5565: URL: https://github.com/apache/hadoop/pull/5565 ### Description of PR When RM starts, because the file corresponding to the `yarn.resourcemanager.nodes.include-path` or `yarn.resourcemanager.nodes.exclude-path` configuration item does not exist,

[GitHub] [hadoop] ashutoshcipher commented on a diff in pull request #5562: YARN-11463. Node Labels root directory creation doesn't have a retry logic

2023-04-18 Thread via GitHub
ashutoshcipher commented on code in PR #5562: URL: https://github.com/apache/hadoop/pull/5562#discussion_r1169877196 ## hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java: ## @@ -137,6 +137,16 @@ private static void

[GitHub] [hadoop] ashutoshcipher commented on a diff in pull request #5562: YARN-11463. Node Labels root directory creation doesn't have a retry logic

2023-04-18 Thread via GitHub
ashutoshcipher commented on code in PR #5562: URL: https://github.com/apache/hadoop/pull/5562#discussion_r1169876664 ## hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java: ## @@ -137,6 +137,16 @@ private static void

[GitHub] [hadoop] ashutoshcipher commented on pull request #5562: YARN-11463. Node Labels root directory creation doesn't have a retry logic

2023-04-18 Thread via GitHub
ashutoshcipher commented on PR #5562: URL: https://github.com/apache/hadoop/pull/5562#issuecomment-1512895864 > Thanks @ashutoshcipher for the update. Apart from two nits it looks good to me. Can you please check the test failure? Thanks @brumi1024 for review. I will look at it.

[GitHub] [hadoop] ashutoshcipher commented on a diff in pull request #5562: YARN-11463. Node Labels root directory creation doesn't have a retry logic

2023-04-18 Thread via GitHub
ashutoshcipher commented on code in PR #5562: URL: https://github.com/apache/hadoop/pull/5562#discussion_r1169876169 ## hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/store/AbstractFSNodeStore.java: ## @@ -65,8 +65,30 @@

[GitHub] [hadoop] brumi1024 commented on a diff in pull request #5562: YARN-11463. Node Labels root directory creation doesn't have a retry logic

2023-04-18 Thread via GitHub
brumi1024 commented on code in PR #5562: URL: https://github.com/apache/hadoop/pull/5562#discussion_r1169804356 ## hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java: ## @@ -137,6 +137,16 @@ private static void

[GitHub] [hadoop] hadoop-yetus commented on pull request #5564: HDFS-16985. delete local block file when FileNotFoundException occurred may lead to missing block.

2023-04-18 Thread via GitHub
hadoop-yetus commented on PR #5564: URL: https://github.com/apache/hadoop/pull/5564#issuecomment-1512750257 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Updated] (HADOOP-18705) hadoop-azure: AzureBlobFileSystem should exclude incompatible credential providers when binding DelegationTokenManagers

2023-04-18 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18705: Affects Version/s: 3.3.5 > hadoop-azure: AzureBlobFileSystem should exclude incompatible

[jira] [Updated] (HADOOP-18705) hadoop-azure: AzureBlobFileSystem should exclude incompatible credential providers when binding DelegationTokenManagers

2023-04-18 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18705: Component/s: fs/azure (was: tools) > hadoop-azure:

[jira] [Commented] (HADOOP-18470) Release hadoop 3.3.5

2023-04-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713496#comment-17713496 ] ASF GitHub Bot commented on HADOOP-18470: - steveloughran merged PR #5558: URL:

[GitHub] [hadoop] steveloughran merged pull request #5558: HADOOP-18470. 3.3.5 Release wrap-up: jdiff files

2023-04-18 Thread via GitHub
steveloughran merged PR #5558: URL: https://github.com/apache/hadoop/pull/5558 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [hadoop] Hexiaoqiao commented on pull request #5564: HDFS-16985. delete local block file when FileNotFoundException occurred may lead to missing block.

2023-04-18 Thread via GitHub
Hexiaoqiao commented on PR #5564: URL: https://github.com/apache/hadoop/pull/5564#issuecomment-1512646630 Agree that we need to protect data for this case. But the current improvement will leave another issue. Considering that DataNode notify NameNode without delete block file here,

[GitHub] [hadoop] hadoop-yetus commented on pull request #5363: YARN-11424. [Federation] Router Supports DeregisterSubCluster.

2023-04-18 Thread via GitHub
hadoop-yetus commented on PR #5363: URL: https://github.com/apache/hadoop/pull/5363#issuecomment-1512630408 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18657) Tune ABFS create() retry logic

2023-04-18 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17713439#comment-17713439 ] ASF GitHub Bot commented on HADOOP-18657: - snvijaya commented on code in PR #5462: URL:

[GitHub] [hadoop] snvijaya commented on a diff in pull request #5462: HADOOP-18657. Tune ABFS create() retry logic

2023-04-18 Thread via GitHub
snvijaya commented on code in PR #5462: URL: https://github.com/apache/hadoop/pull/5462#discussion_r1131060581 ## hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java: ## @@ -621,37 +622,57 @@ private AbfsRestOperation

[GitHub] [hadoop] smarthanwang commented on pull request #5564: HDFS-16985. delete local block file when FileNotFoundException occurred may lead to missing block.

2023-04-18 Thread via GitHub
smarthanwang commented on PR #5564: URL: https://github.com/apache/hadoop/pull/5564#issuecomment-1512506592 > If there's only one replica, and AWS EBS fails, doesn't it mean the file is not recoverable anyway? 1. The EBS not really failed, it may recover soon. 2. The EBS really