[GitHub] [hadoop] virajjasani commented on pull request #5523: HDFS-16967. RBF: File based state stores should allow concurrent access to the records

2023-03-30 Thread via GitHub
virajjasani commented on PR #5523: URL: https://github.com/apache/hadoop/pull/5523#issuecomment-1491325194 Default mode: https://user-images.githubusercontent.com/34790606/229032556-3603807e-94de-48f8-87bc-831f4519ed11.png;>

[GitHub] [hadoop] rdingankar commented on pull request #5495: HDFS-16949 Introduce inverse quantiles for metrics where higher numer…

2023-03-30 Thread via GitHub
rdingankar commented on PR #5495: URL: https://github.com/apache/hadoop/pull/5495#issuecomment-1491253601 The failed UTs seem flaky and unrelated to the change in this PR. They pass locally. @goiri Can you please help merging the PR? -- This is an automated message from the Apache Git

[jira] [Commented] (HADOOP-18680) Insufficient heap during full test runs in Docker container.

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17707100#comment-17707100 ] ASF GitHub Bot commented on HADOOP-18680: - hadoop-yetus commented on PR #5522: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5522: HADOOP-18680: Insufficient heap during full test runs in Docker container.

2023-03-30 Thread via GitHub
hadoop-yetus commented on PR #5522: URL: https://github.com/apache/hadoop/pull/5522#issuecomment-1491249469 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] cnauroth commented on a diff in pull request #5520: HDFS-16965. Add switch to decide whether to enable native codec.

2023-03-30 Thread via GitHub
cnauroth commented on code in PR #5520: URL: https://github.com/apache/hadoop/pull/5520#discussion_r1153980763 ## hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/CodecUtil.java: ## @@ -170,8 +174,14 @@ private static String[] getRawCoderNames(

[jira] [Commented] (HADOOP-18684) Fix S3A filesystem such that the scheme matches the URI scheme

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17707098#comment-17707098 ] ASF GitHub Bot commented on HADOOP-18684: - cnauroth commented on code in PR #5521: URL:

[GitHub] [hadoop] cnauroth commented on a diff in pull request #5521: HADOOP-18684: Fix S3A filesystem such that the scheme matches the URI scheme

2023-03-30 Thread via GitHub
cnauroth commented on code in PR #5521: URL: https://github.com/apache/hadoop/pull/5521#discussion_r1153974579 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/InternalConstants.java: ## @@ -182,4 +182,9 @@ private InternalConstants() {

[jira] [Commented] (HADOOP-18680) Insufficient heap during full test runs in Docker container.

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17707093#comment-17707093 ] ASF GitHub Bot commented on HADOOP-18680: - hadoop-yetus commented on PR #5522: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5522: HADOOP-18680: Insufficient heap during full test runs in Docker container.

2023-03-30 Thread via GitHub
hadoop-yetus commented on PR #5522: URL: https://github.com/apache/hadoop/pull/5522#issuecomment-1491227005 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at

[jira] [Commented] (HADOOP-18680) Insufficient heap during full test runs in Docker container.

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17707092#comment-17707092 ] ASF GitHub Bot commented on HADOOP-18680: - hadoop-yetus commented on PR #5522: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5522: HADOOP-18680: Insufficient heap during full test runs in Docker container.

2023-03-30 Thread via GitHub
hadoop-yetus commented on PR #5522: URL: https://github.com/apache/hadoop/pull/5522#issuecomment-1491225676 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18680) Insufficient heap during full test runs in Docker container.

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17707081#comment-17707081 ] ASF GitHub Bot commented on HADOOP-18680: - hadoop-yetus commented on PR #5522: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5522: HADOOP-18680: Insufficient heap during full test runs in Docker container.

2023-03-30 Thread via GitHub
hadoop-yetus commented on PR #5522: URL: https://github.com/apache/hadoop/pull/5522#issuecomment-1491166976 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5495: HDFS-16949 Introduce inverse quantiles for metrics where higher numer…

2023-03-30 Thread via GitHub
hadoop-yetus commented on PR #5495: URL: https://github.com/apache/hadoop/pull/5495#issuecomment-1491146475 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5523: HDFS-16967. RBF: File based state stores should allow concurrent access to the records

2023-03-30 Thread via GitHub
hadoop-yetus commented on PR #5523: URL: https://github.com/apache/hadoop/pull/5523#issuecomment-1491141117 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18680) Insufficient heap during full test runs in Docker container.

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17707073#comment-17707073 ] ASF GitHub Bot commented on HADOOP-18680: - hadoop-yetus commented on PR #5522: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5522: HADOOP-18680: Insufficient heap during full test runs in Docker container.

2023-03-30 Thread via GitHub
hadoop-yetus commented on PR #5522: URL: https://github.com/apache/hadoop/pull/5522#issuecomment-1491125289 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] cnauroth commented on a diff in pull request #5519: MAPREDUCE-7435. Manifest Committer OOM on abfs

2023-03-30 Thread via GitHub
cnauroth commented on code in PR #5519: URL: https://github.com/apache/hadoop/pull/5519#discussion_r1153857994 ##

[GitHub] [hadoop] virajjasani commented on pull request #5523: HDFS-16967. RBF: File based state stores should allow concurrent access to the records

2023-03-30 Thread via GitHub
virajjasani commented on PR #5523: URL: https://github.com/apache/hadoop/pull/5523#issuecomment-1491057858 Based on one of the testing data points, for the same num of mount table records to be loaded in the cache, avg time taken by default is ~1500 ms whereas with concurrent mode, it goes

[GitHub] [hadoop] virajjasani commented on pull request #5523: HDFS-16967. RBF: File based state stores should allow concurrent access to the records

2023-03-30 Thread via GitHub
virajjasani commented on PR #5523: URL: https://github.com/apache/hadoop/pull/5523#issuecomment-1491056033 @goiri @ZanderXu @tasanuma could you please review this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [hadoop] virajjasani opened a new pull request, #5523: HDFS-16967. RBF: File based state stores should allow concurrent access to the records

2023-03-30 Thread via GitHub
virajjasani opened a new pull request, #5523: URL: https://github.com/apache/hadoop/pull/5523 File based state store implementations (StateStoreFileImpl and StateStoreFileSystemImpl) should allow updating as well as reading of the state store records concurrently rather than serially.

[GitHub] [hadoop] rdingankar commented on a diff in pull request #5495: HDFS-16949 Introduce inverse quantiles for metrics where higher numer…

2023-03-30 Thread via GitHub
rdingankar commented on code in PR #5495: URL: https://github.com/apache/hadoop/pull/5495#discussion_r1153853239 ## hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/util/TestSampleQuantiles.java: ## @@ -37,8 +37,10 @@ public class TestSampleQuantiles

[GitHub] [hadoop] rdingankar commented on a diff in pull request #5495: HDFS-16949 Introduce inverse quantiles for metrics where higher numer…

2023-03-30 Thread via GitHub
rdingankar commented on code in PR #5495: URL: https://github.com/apache/hadoop/pull/5495#discussion_r1153852005 ## hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/util/TestSampleQuantiles.java: ## @@ -92,22 +94,21 @@ public void testClear() throws

[jira] [Commented] (HADOOP-18680) Insufficient heap during full test runs in Docker container.

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17707055#comment-17707055 ] ASF GitHub Bot commented on HADOOP-18680: - cnauroth commented on code in PR #5511: URL:

[GitHub] [hadoop] goiri commented on a diff in pull request #5495: HDFS-16949 Introduce inverse quantiles for metrics where higher numer…

2023-03-30 Thread via GitHub
goiri commented on code in PR #5495: URL: https://github.com/apache/hadoop/pull/5495#discussion_r1153848968 ## hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/util/TestSampleQuantiles.java: ## @@ -37,8 +37,10 @@ public class TestSampleQuantiles {

[GitHub] [hadoop] cnauroth commented on a diff in pull request #5511: HADOOP-18680: Insufficient heap during full test runs in Docker container.

2023-03-30 Thread via GitHub
cnauroth commented on code in PR #5511: URL: https://github.com/apache/hadoop/pull/5511#discussion_r1153849194 ## dev-support/docker/Dockerfile_aarch64: ## @@ -144,7 +144,7 @@ RUN npm install -g bower@1.8.8 ### # Avoid out of memory errors in builds ### -ENV MAVEN_OPTS

[GitHub] [hadoop] hadoop-yetus commented on pull request #5495: HDFS-16949 Introduce inverse quantiles for metrics where higher numer…

2023-03-30 Thread via GitHub
hadoop-yetus commented on PR #5495: URL: https://github.com/apache/hadoop/pull/5495#issuecomment-1491048324 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18680) Insufficient heap during full test runs in Docker container.

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17707053#comment-17707053 ] ASF GitHub Bot commented on HADOOP-18680: - cnauroth opened a new pull request, #5522: URL:

[GitHub] [hadoop] cnauroth opened a new pull request, #5522: HADOOP-18680: Insufficient heap during full test runs in Docker container.

2023-03-30 Thread via GitHub
cnauroth opened a new pull request, #5522: URL: https://github.com/apache/hadoop/pull/5522 ### Description of PR The `MAVEN_OPTS` max heap size setting is different between Dockerfile vs. Dockerfile_aarch64 on trunk. The Dockerfile setting is larger to avoid running out of memory

[jira] [Updated] (HADOOP-18680) Insufficient heap during full test runs in Docker container.

2023-03-30 Thread Chris Nauroth (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-18680: --- Description: During verification of releases on the 3.3 line, I often run out of heap

[jira] [Commented] (HADOOP-18680) Insufficient heap during full test runs in Docker container on branch-3.3.

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706989#comment-17706989 ] ASF GitHub Bot commented on HADOOP-18680: - ayushtkn commented on code in PR #5511: URL:

[GitHub] [hadoop] ayushtkn commented on a diff in pull request #5511: HADOOP-18680: Insufficient heap during full test runs in Docker container on branch-3.3.

2023-03-30 Thread via GitHub
ayushtkn commented on code in PR #5511: URL: https://github.com/apache/hadoop/pull/5511#discussion_r1153646744 ## dev-support/docker/Dockerfile_aarch64: ## @@ -144,7 +144,7 @@ RUN npm install -g bower@1.8.8 ### # Avoid out of memory errors in builds ### -ENV MAVEN_OPTS

[jira] [Commented] (HADOOP-18680) Insufficient heap during full test runs in Docker container on branch-3.3.

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706988#comment-17706988 ] ASF GitHub Bot commented on HADOOP-18680: - ayushtkn commented on code in PR #5511: URL:

[GitHub] [hadoop] ayushtkn commented on a diff in pull request #5511: HADOOP-18680: Insufficient heap during full test runs in Docker container on branch-3.3.

2023-03-30 Thread via GitHub
ayushtkn commented on code in PR #5511: URL: https://github.com/apache/hadoop/pull/5511#discussion_r1153646744 ## dev-support/docker/Dockerfile_aarch64: ## @@ -144,7 +144,7 @@ RUN npm install -g bower@1.8.8 ### # Avoid out of memory errors in builds ### -ENV MAVEN_OPTS

[GitHub] [hadoop] hadoop-yetus commented on pull request #5363: YARN-11424. [Federation] Router Supports DeregisterSubCluster.

2023-03-30 Thread via GitHub
hadoop-yetus commented on PR #5363: URL: https://github.com/apache/hadoop/pull/5363#issuecomment-1490667779 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18684) Fix S3A filesystem such that the scheme matches the URI scheme

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706975#comment-17706975 ] ASF GitHub Bot commented on HADOOP-18684: - hadoop-yetus commented on PR #5521: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5521: HADOOP-18684: Fix S3A filesystem such that the scheme matches the URI scheme

2023-03-30 Thread via GitHub
hadoop-yetus commented on PR #5521: URL: https://github.com/apache/hadoop/pull/5521#issuecomment-1490651740 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18684) Fix S3A filesystem such that the scheme matches the URI scheme

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706971#comment-17706971 ] ASF GitHub Bot commented on HADOOP-18684: - hadoop-yetus commented on PR #5521: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5521: HADOOP-18684: Fix S3A filesystem such that the scheme matches the URI scheme

2023-03-30 Thread via GitHub
hadoop-yetus commented on PR #5521: URL: https://github.com/apache/hadoop/pull/5521#issuecomment-1490635342 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18684) Fix S3A filesystem such that the scheme matches the URI scheme

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706970#comment-17706970 ] ASF GitHub Bot commented on HADOOP-18684: - hadoop-yetus commented on PR #5521: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5521: HADOOP-18684: Fix S3A filesystem such that the scheme matches the URI scheme

2023-03-30 Thread via GitHub
hadoop-yetus commented on PR #5521: URL: https://github.com/apache/hadoop/pull/5521#issuecomment-1490632200 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5520: HDFS-16965. Add switch to decide whether to enable native codec.

2023-03-30 Thread via GitHub
hadoop-yetus commented on PR #5520: URL: https://github.com/apache/hadoop/pull/5520#issuecomment-1490629400 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18684) Fix S3A filesystem such that the scheme matches the URI scheme

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706964#comment-17706964 ] ASF GitHub Bot commented on HADOOP-18684: - hadoop-yetus commented on PR #5521: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5521: HADOOP-18684: Fix S3A filesystem such that the scheme matches the URI scheme

2023-03-30 Thread via GitHub
hadoop-yetus commented on PR #5521: URL: https://github.com/apache/hadoop/pull/5521#issuecomment-1490620563 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18684) Fix S3A filesystem such that the scheme matches the URI scheme

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706958#comment-17706958 ] ASF GitHub Bot commented on HADOOP-18684: - hadoop-yetus commented on PR #5521: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5521: HADOOP-18684: Fix S3A filesystem such that the scheme matches the URI scheme

2023-03-30 Thread via GitHub
hadoop-yetus commented on PR #5521: URL: https://github.com/apache/hadoop/pull/5521#issuecomment-1490603596 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18671) Add recoverLease(), setSafeMode(), isFileClosed() APIs to FileSystem

2023-03-30 Thread Tak-Lon (Stephen) Wu (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706942#comment-17706942 ] Tak-Lon (Stephen) Wu commented on HADOOP-18671: --- I'm going to give a try, will create the

[GitHub] [hadoop] hadoop-yetus commented on pull request #4963: YARN-11326. [Federation] Add RM FederationStateStoreService Metrics.

2023-03-30 Thread via GitHub
hadoop-yetus commented on PR #4963: URL: https://github.com/apache/hadoop/pull/4963#issuecomment-1490521782 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18565) AWS SDK V2 - Complete outstanding items

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706902#comment-17706902 ] ASF GitHub Bot commented on HADOOP-18565: - ahmarsuhail commented on PR #5421: URL:

[GitHub] [hadoop] ahmarsuhail commented on pull request #5421: HADOOP-18565. Completes outstanding items for the SDK V2 upgrade.

2023-03-30 Thread via GitHub
ahmarsuhail commented on PR #5421: URL: https://github.com/apache/hadoop/pull/5421#issuecomment-1490355464 Add the amazon.cn check where? haven't added `Endpoint is not set || Endpoint is set && ends in amazonaws.com || ARN is set` in this PR (if that is where you meant) -- This is an

[jira] [Commented] (HADOOP-18565) AWS SDK V2 - Complete outstanding items

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706899#comment-17706899 ] ASF GitHub Bot commented on HADOOP-18565: - steveloughran commented on PR #5421: URL:

[GitHub] [hadoop] steveloughran commented on pull request #5421: HADOOP-18565. Completes outstanding items for the SDK V2 upgrade.

2023-03-30 Thread via GitHub
steveloughran commented on PR #5421: URL: https://github.com/apache/hadoop/pull/5421#issuecomment-1490348322 w.r.t merging, add the amazon.cn check so it doesn't get forgotten about. then we should be good to mergel -- This is an automated message from the Apache Git Service. To respond

[jira] [Commented] (HADOOP-18565) AWS SDK V2 - Complete outstanding items

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706897#comment-17706897 ] ASF GitHub Bot commented on HADOOP-18565: - steveloughran commented on PR #5421: URL:

[GitHub] [hadoop] steveloughran commented on pull request #5421: HADOOP-18565. Completes outstanding items for the SDK V2 upgrade.

2023-03-30 Thread via GitHub
steveloughran commented on PR #5421: URL: https://github.com/apache/hadoop/pull/5421#issuecomment-1490346831 i don't know about those third party stores; someone in your sdk team probably knows better there. I think generally it is just account + secret, unless something like

[GitHub] [hadoop] slfan1989 commented on pull request #5491: HDFS-16951. Add description of GETSERVERDEFAULTS to WebHDFS doc

2023-03-30 Thread via GitHub
slfan1989 commented on PR #5491: URL: https://github.com/apache/hadoop/pull/5491#issuecomment-1490331247 LGTM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[jira] [Updated] (HADOOP-18647) x-ms-client-request-id to have some way that identifies retry of an API.

2023-03-30 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18647: Fix Version/s: 3.3.9 > x-ms-client-request-id to have some way that identifies retry of

[jira] [Commented] (HADOOP-18647) x-ms-client-request-id to have some way that identifies retry of an API.

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706893#comment-17706893 ] ASF GitHub Bot commented on HADOOP-18647: - steveloughran commented on PR #5498: URL:

[jira] [Commented] (HADOOP-18647) x-ms-client-request-id to have some way that identifies retry of an API.

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706892#comment-17706892 ] ASF GitHub Bot commented on HADOOP-18647: - steveloughran merged PR #5498: URL:

[GitHub] [hadoop] steveloughran commented on pull request #5498: HADOOP-18647. x-ms-client-request-id to have some way that identifies retry of an API. (#5437)

2023-03-30 Thread via GitHub
steveloughran commented on PR #5498: URL: https://github.com/apache/hadoop/pull/5498#issuecomment-1490303264 +1, merged -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [hadoop] steveloughran merged pull request #5498: HADOOP-18647. x-ms-client-request-id to have some way that identifies retry of an API. (#5437)

2023-03-30 Thread via GitHub
steveloughran merged PR #5498: URL: https://github.com/apache/hadoop/pull/5498 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Commented] (HADOOP-18012) ABFS: Enable config controlled ETag check for Rename idempotency

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706887#comment-17706887 ] ASF GitHub Bot commented on HADOOP-18012: - steveloughran commented on PR #5488: URL:

[GitHub] [hadoop] steveloughran commented on pull request #5488: HADOOP-18012. ABFS: Using Source Path eTags for Rename Idemptonency checks

2023-03-30 Thread via GitHub
steveloughran commented on PR #5488: URL: https://github.com/apache/hadoop/pull/5488#issuecomment-1490292881 +1 pending the merge and checkstyle issues ``` ./hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClientRenameResult.java:64: return

[jira] [Commented] (HADOOP-18012) ABFS: Enable config controlled ETag check for Rename idempotency

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706885#comment-17706885 ] ASF GitHub Bot commented on HADOOP-18012: - steveloughran commented on code in PR #5488: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5420: YARN-11442. Refactor FederationInterceptorREST Code.

2023-03-30 Thread via GitHub
hadoop-yetus commented on PR #5420: URL: https://github.com/apache/hadoop/pull/5420#issuecomment-1490289178 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] steveloughran commented on a diff in pull request #5488: HADOOP-18012. ABFS: Using Source Path eTags for Rename Idemptonency checks

2023-03-30 Thread via GitHub
steveloughran commented on code in PR #5488: URL: https://github.com/apache/hadoop/pull/5488#discussion_r1153240354 ## hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java: ## @@ -503,27 +512,50 @@ public AbfsRestOperation

[jira] [Updated] (HADOOP-18685) Amazon S3 disabling ACLs on all new buckets

2023-03-30 Thread Daniel Carl Jones (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Carl Jones updated HADOOP-18685: --- Description: In April 2023, Amazon S3 will be disabling ACLs by default on *all*

[jira] [Commented] (HADOOP-18685) Amazon S3 disabling ACLs on all new buckets

2023-03-30 Thread Daniel Carl Jones (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706883#comment-17706883 ] Daniel Carl Jones commented on HADOOP-18685: This would already be tripping up developers,

[jira] [Commented] (HADOOP-18197) Update protobuf 3.7.1 to a version without CVE-2021-22569

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706870#comment-17706870 ] ASF GitHub Bot commented on HADOOP-18197: - steveloughran commented on PR #4418: URL:

[GitHub] [hadoop] steveloughran commented on pull request #4418: HADOOP-18197. Upgrade protobuf to 3.21.7 (through upgraded hadoop-shaded-protobuf jar)

2023-03-30 Thread via GitHub
steveloughran commented on PR #4418: URL: https://github.com/apache/hadoop/pull/4418#issuecomment-1490261364 @xizhu-mstr @tooptoop4 i'm not actively working on this; too many pressing issues and after getting 3.3.5 out the door I'm catching up with the internal stuff. Either of you two

[jira] [Commented] (HADOOP-18197) Update protobuf 3.7.1 to a version without CVE-2021-22569

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706867#comment-17706867 ] ASF GitHub Bot commented on HADOOP-18197: - steveloughran commented on code in PR #4418: URL:

[GitHub] [hadoop] steveloughran commented on a diff in pull request #4418: HADOOP-18197. Upgrade protobuf to 3.21.7 (through upgraded hadoop-shaded-protobuf jar)

2023-03-30 Thread via GitHub
steveloughran commented on code in PR #4418: URL: https://github.com/apache/hadoop/pull/4418#discussion_r1153223024 ## BUILDING.txt: ## @@ -403,10 +403,10 @@ Installing required dependencies for clean install of macOS 10.14: * Install native libraries, only openssl is

[jira] [Created] (HADOOP-18685) Amazon S3 changing ACL defaults

2023-03-30 Thread Daniel Carl Jones (Jira)
Daniel Carl Jones created HADOOP-18685: -- Summary: Amazon S3 changing ACL defaults Key: HADOOP-18685 URL: https://issues.apache.org/jira/browse/HADOOP-18685 Project: Hadoop Common Issue

[jira] [Updated] (HADOOP-18685) Amazon S3 disabling ACLs on all new buckets

2023-03-30 Thread Daniel Carl Jones (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Carl Jones updated HADOOP-18685: --- Summary: Amazon S3 disabling ACLs on all new buckets (was: Amazon S3 changing ACL

[GitHub] [hadoop] hadoop-yetus commented on pull request #5495: HDFS-16949 Introduce inverse quantiles for metrics where higher numer…

2023-03-30 Thread via GitHub
hadoop-yetus commented on PR #5495: URL: https://github.com/apache/hadoop/pull/5495#issuecomment-1490184577 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5520: HDFS-16965. Add switch to decide whether to enable native codec.

2023-03-30 Thread via GitHub
hadoop-yetus commented on PR #5520: URL: https://github.com/apache/hadoop/pull/5520#issuecomment-1490165934 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5495: HDFS-16949 Introduce inverse quantiles for metrics where higher numer…

2023-03-30 Thread via GitHub
hadoop-yetus commented on PR #5495: URL: https://github.com/apache/hadoop/pull/5495#issuecomment-1490153112 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18684) Fix S3A filesystem such that the scheme matches the URI scheme

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706843#comment-17706843 ] ASF GitHub Bot commented on HADOOP-18684: - hadoop-yetus commented on PR #5521: URL:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5521: HADOOP-18684: Fix S3A filesystem such that the scheme matches the URI scheme

2023-03-30 Thread via GitHub
hadoop-yetus commented on PR #5521: URL: https://github.com/apache/hadoop/pull/5521#issuecomment-1490137673 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5420: YARN-11442. Refactor FederationInterceptorREST Code.

2023-03-30 Thread via GitHub
hadoop-yetus commented on PR #5420: URL: https://github.com/apache/hadoop/pull/5420#issuecomment-1490106434 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] zhtttylz commented on pull request #5517: HDFS-16952. Support getLinkTarget API in WebHDFS

2023-03-30 Thread via GitHub
zhtttylz commented on PR #5517: URL: https://github.com/apache/hadoop/pull/5517#issuecomment-1490101718 @slfan1989 I've updated the PR based on your suggestion, could you review this please. thanks. -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [hadoop] hadoop-yetus commented on pull request #5517: HDFS-16952. Support getLinkTarget API in WebHDFS

2023-03-30 Thread via GitHub
hadoop-yetus commented on PR #5517: URL: https://github.com/apache/hadoop/pull/5517#issuecomment-1490079978 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18684) Fix S3A filesystem such that the scheme matches the URI scheme

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706817#comment-17706817 ] ASF GitHub Bot commented on HADOOP-18684: - steveloughran commented on code in PR #5521: URL:

[GitHub] [hadoop] steveloughran commented on a diff in pull request #5521: HADOOP-18684: Fix S3A filesystem such that the scheme matches the URI scheme

2023-03-30 Thread via GitHub
steveloughran commented on code in PR #5521: URL: https://github.com/apache/hadoop/pull/5521#discussion_r1153027000 ## hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AUrlScheme.java: ## @@ -0,0 +1,31 @@ +package org.apache.hadoop.fs.s3a; + +import

[jira] [Updated] (HADOOP-18684) Fix S3A filesystem such that the scheme matches the URI scheme

2023-03-30 Thread Harshit Gupta (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harshit Gupta updated HADOOP-18684: --- Affects Version/s: 3.3.5 > Fix S3A filesystem such that the scheme matches the URI scheme

[jira] [Commented] (HADOOP-18684) Fix S3A filesystem such that the scheme matches the URI scheme

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706809#comment-17706809 ] ASF GitHub Bot commented on HADOOP-18684: - HarshitGupta11 opened a new pull request, #5521:

[jira] [Updated] (HADOOP-18684) Fix S3A filesystem such that the scheme matches the URI scheme

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-18684: Labels: pull-request-available (was: ) > Fix S3A filesystem such that the scheme

[GitHub] [hadoop] HarshitGupta11 opened a new pull request, #5521: HADOOP-18684: Fix S3A filesystem such that the scheme matches the URI scheme

2023-03-30 Thread via GitHub
HarshitGupta11 opened a new pull request, #5521: URL: https://github.com/apache/hadoop/pull/5521 ### Description of PR Certain codepaths use the FileContext API's to perform FS based operations such as yarn log aggregations. While trying to reuse the S3A connector for GCS based

[GitHub] [hadoop] hadoop-yetus commented on pull request #5495: HDFS-16949 Introduce inverse quantiles for metrics where higher numer…

2023-03-30 Thread via GitHub
hadoop-yetus commented on PR #5495: URL: https://github.com/apache/hadoop/pull/5495#issuecomment-1489978632 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18197) Update protobuf 3.7.1 to a version without CVE-2021-22569

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706779#comment-17706779 ] ASF GitHub Bot commented on HADOOP-18197: - xizhu-mstr commented on code in PR #4418: URL:

[GitHub] [hadoop] xizhu-mstr commented on a diff in pull request #4418: HADOOP-18197. Upgrade protobuf to 3.21.7 (through upgraded hadoop-shaded-protobuf jar)

2023-03-30 Thread via GitHub
xizhu-mstr commented on code in PR #4418: URL: https://github.com/apache/hadoop/pull/4418#discussion_r1152934268 ## BUILDING.txt: ## @@ -403,10 +403,10 @@ Installing required dependencies for clean install of macOS 10.14: * Install native libraries, only openssl is required

[jira] [Commented] (HADOOP-18197) Update protobuf 3.7.1 to a version without CVE-2021-22569

2023-03-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706777#comment-17706777 ] ASF GitHub Bot commented on HADOOP-18197: - xizhu-mstr commented on code in PR #4418: URL:

[GitHub] [hadoop] xizhu-mstr commented on a diff in pull request #4418: HADOOP-18197. Upgrade protobuf to 3.21.7 (through upgraded hadoop-shaded-protobuf jar)

2023-03-30 Thread via GitHub
xizhu-mstr commented on code in PR #4418: URL: https://github.com/apache/hadoop/pull/4418#discussion_r1152934268 ## BUILDING.txt: ## @@ -403,10 +403,10 @@ Installing required dependencies for clean install of macOS 10.14: * Install native libraries, only openssl is required

[GitHub] [hadoop] WYB489341394 commented on pull request #5520: HDFS-16965. Add switch to decide whether to enable native codec.

2023-03-30 Thread via GitHub
WYB489341394 commented on PR #5520: URL: https://github.com/apache/hadoop/pull/5520#issuecomment-1489885854 JIRA: [HDFS-16965](https://issues.apache.org/jira/browse/HDFS-16965). Add switch to decide whether to enable native codec. -- This is an automated message from the Apache Git

[GitHub] [hadoop] WYB489341394 opened a new pull request, #5520: HDFS-16965. Add switch to decide whether to enable native codec.

2023-03-30 Thread via GitHub
WYB489341394 opened a new pull request, #5520: URL: https://github.com/apache/hadoop/pull/5520 ### Description of PR ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g.

[GitHub] [hadoop] hadoop-yetus commented on pull request #5363: YARN-11424. [Federation] Router Supports DeregisterSubCluster.

2023-03-30 Thread via GitHub
hadoop-yetus commented on PR #5363: URL: https://github.com/apache/hadoop/pull/5363#issuecomment-1489864292 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #4687: YARN-10846. Add dispatcher metrics to NM.

2023-03-30 Thread via GitHub
hadoop-yetus commented on PR #4687: URL: https://github.com/apache/hadoop/pull/4687#issuecomment-1489758181 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[GitHub] [hadoop] hadoop-yetus commented on pull request #5495: HDFS-16949 Introduce inverse quantiles for metrics where higher numer…

2023-03-30 Thread via GitHub
hadoop-yetus commented on PR #5495: URL: https://github.com/apache/hadoop/pull/5495#issuecomment-1489755320 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |