[GitHub] [hadoop] hadoop-yetus commented on pull request #5879: HDFS-17130. Blocks on IN_MAINTENANCE DNs should be sorted properly in LocatedBlocks.

2023-07-27 Thread via GitHub
hadoop-yetus commented on PR #5879: URL: https://github.com/apache/hadoop/pull/5879#issuecomment-1655126417 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

[GitHub] [hadoop] yangjiandan commented on pull request #5890: YARN-11538. CS UI: queue filter do not work as expected when submitti…

2023-07-27 Thread via GitHub
yangjiandan commented on PR #5890: URL: https://github.com/apache/hadoop/pull/5890#issuecomment-1655113232 @slfan1989 Thanks for your help in reviewing my code and helping me resolve the issues. Your feedback was invaluable and has greatly improved my contribution. -- This is an automate

[jira] [Commented] (HADOOP-18820) AWS SDK v2: make the v1 bridging support optional

2023-07-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17748419#comment-17748419 ] ASF GitHub Bot commented on HADOOP-18820: - hadoop-yetus commented on PR #5872:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5872: HADOOP-18820. Cut AWS v1 support

2023-07-27 Thread via GitHub
hadoop-yetus commented on PR #5872: URL: https://github.com/apache/hadoop/pull/5872#issuecomment-1655062477 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

[GitHub] [hadoop] slfan1989 merged pull request #5810: YARN-7707. BackPort [GPG] Policy generator framework.

2023-07-27 Thread via GitHub
slfan1989 merged PR #5810: URL: https://github.com/apache/hadoop/pull/5810 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@had

[GitHub] [hadoop] slfan1989 commented on pull request #5810: YARN-7707. BackPort [GPG] Policy generator framework.

2023-07-27 Thread via GitHub
slfan1989 commented on PR #5810: URL: https://github.com/apache/hadoop/pull/5810#issuecomment-1655053097 @goiri I will merge this pr into the trunk branch and then follow up on BackPort YARN-8934. Thanks again for your help in reviewing the code! -- This is an automated message from the A

[GitHub] [hadoop] slfan1989 commented on pull request #5897: HDFS-17128. Updating SQLDelegationTokenSecretManager to use LoadingCa…

2023-07-27 Thread via GitHub
slfan1989 commented on PR #5897: URL: https://github.com/apache/hadoop/pull/5897#issuecomment-1655049316 Good Catch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

[GitHub] [hadoop] slfan1989 commented on pull request #5890: YARN-11538. CS UI: queue filter do not work as expected when submitti…

2023-07-27 Thread via GitHub
slfan1989 commented on PR #5890: URL: https://github.com/apache/hadoop/pull/5890#issuecomment-1655048767 LGTM. If there is no suggestion from other partners, it will be merged to the trunk after 2-3 days. -- This is an automated message from the Apache Git Service. To respond to the messa

[GitHub] [hadoop] haiyang1987 commented on pull request #5876: HDFS-17116. RBF: Update invoke millisecond time as monotonicNow() in RouterSafemodeService

2023-07-27 Thread via GitHub
haiyang1987 commented on PR #5876: URL: https://github.com/apache/hadoop/pull/5876#issuecomment-1655034285 The failed unit test seems unrelated to the change, I will follow up on this UT failure issue and create a new issue to solve it -- This is an automated message from the Apache Git S

[GitHub] [hadoop] hadoop-yetus commented on pull request #5890: YARN-11538. CS UI: queue filter do not work as expected when submitti…

2023-07-27 Thread via GitHub
hadoop-yetus commented on PR #5890: URL: https://github.com/apache/hadoop/pull/5890#issuecomment-1654999213 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reex

[GitHub] [hadoop] hadoop-yetus commented on pull request #5897: HDFS-17128. Updating SQLDelegationTokenSecretManager to use LoadingCa…

2023-07-27 Thread via GitHub
hadoop-yetus commented on PR #5897: URL: https://github.com/apache/hadoop/pull/5897#issuecomment-1654950638 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

[GitHub] [hadoop] Tre2878 commented on a diff in pull request #5855: HDFS-17093. In the case of all datanodes sending FBR when the namenode restarts (large clusters), there is an issue with incomplete

2023-07-27 Thread via GitHub
Tre2878 commented on code in PR #5855: URL: https://github.com/apache/hadoop/pull/5855#discussion_r1277059268 ## hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockReportLease.java: ## @@ -269,4 +272,88 @@ private StorageBlockRep

[jira] [Commented] (HADOOP-18817) Upgrade version of aws-java-sdk-bundle to 1.12.368 avoid verify error.

2023-07-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17748378#comment-17748378 ] ASF GitHub Bot commented on HADOOP-18817: - gp1314 commented on PR #5867: URL: h

[GitHub] [hadoop] gp1314 commented on pull request #5867: HADOOP-18817.Upgrade version of aws-java-sdk-bundle to 1.12.368 avoid verify error.

2023-07-27 Thread via GitHub
gp1314 commented on PR #5867: URL: https://github.com/apache/hadoop/pull/5867#issuecomment-1654901930 My own repository aws-java-sdk-bundle was wrong causing the problem -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [hadoop] zhangshuyan0 commented on pull request #5889: HDFS-17123. Sort datanodeStorages when generating StorageBlockReport[] in method BPServiceActor#blockReport for future convenience

2023-07-27 Thread via GitHub
zhangshuyan0 commented on PR #5889: URL: https://github.com/apache/hadoop/pull/5889#issuecomment-1654876600 Anyway, the meta info that a datanode cat get is very limited, I think we should try to solve this problem in NameNode. -- This is an automated message from the Apache Git Service.

[GitHub] [hadoop] zhangshuyan0 commented on a diff in pull request #5855: HDFS-17093. In the case of all datanodes sending FBR when the namenode restarts (large clusters), there is an issue with incom

2023-07-27 Thread via GitHub
zhangshuyan0 commented on code in PR #5855: URL: https://github.com/apache/hadoop/pull/5855#discussion_r1277008543 ## hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockReportLease.java: ## @@ -269,4 +272,88 @@ private StorageBlo

[GitHub] [hadoop] hadoop-yetus commented on pull request #5714: HDFS-17026. RBF: NamenodeHeartbeatService should update JMX report with configurable frequency

2023-07-27 Thread via GitHub
hadoop-yetus commented on PR #5714: URL: https://github.com/apache/hadoop/pull/5714#issuecomment-1654872092 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reex

[GitHub] [hadoop] zhangshuyan0 commented on pull request #5889: HDFS-17123. Sort datanodeStorages when generating StorageBlockReport[] in method BPServiceActor#blockReport for future convenience

2023-07-27 Thread via GitHub
zhangshuyan0 commented on PR #5889: URL: https://github.com/apache/hadoop/pull/5889#issuecomment-1654868198 The comment below can explain what I mean: https://github.com/apache/hadoop/blob/6dafd53626409a568464281f01af47a9ee85432b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/ha

[jira] [Commented] (HADOOP-18820) AWS SDK v2: make the v1 bridging support optional

2023-07-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17748364#comment-17748364 ] ASF GitHub Bot commented on HADOOP-18820: - hadoop-yetus commented on PR #5872:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5872: HADOOP-18820. Cut AWS v1 support

2023-07-27 Thread via GitHub
hadoop-yetus commented on PR #5872: URL: https://github.com/apache/hadoop/pull/5872#issuecomment-1654848382 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

[GitHub] [hadoop] hchaverri opened a new pull request, #5897: HDFS-17128. Updating SQLDelegationTokenSecretManager to use LoadingCa…

2023-07-27 Thread via GitHub
hchaverri opened a new pull request, #5897: URL: https://github.com/apache/hadoop/pull/5897 …che so tokens are updated frequently ### Description of PR The SQLDelegationTokenSecretManager is used by RBF to store a higher volume of tokens than supported by Zookeeper. Current

[GitHub] [hadoop] hchaverri closed pull request #5714: HDFS-17026. RBF: NamenodeHeartbeatService should update JMX report with configurable frequency

2023-07-27 Thread via GitHub
hchaverri closed pull request #5714: HDFS-17026. RBF: NamenodeHeartbeatService should update JMX report with configurable frequency URL: https://github.com/apache/hadoop/pull/5714 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

[jira] [Commented] (HADOOP-18207) Introduce hadoop-logging module

2023-07-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17748344#comment-17748344 ] ASF GitHub Bot commented on HADOOP-18207: - hadoop-yetus commented on PR #5717:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5717: HADOOP-18207. Introduce hadoop-logging module

2023-07-27 Thread via GitHub
hadoop-yetus commented on PR #5717: URL: https://github.com/apache/hadoop/pull/5717#issuecomment-1654696619 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

[jira] [Commented] (HADOOP-18207) Introduce hadoop-logging module

2023-07-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17748343#comment-17748343 ] ASF GitHub Bot commented on HADOOP-18207: - hadoop-yetus commented on PR #5717:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5717: HADOOP-18207. Introduce hadoop-logging module

2023-07-27 Thread via GitHub
hadoop-yetus commented on PR #5717: URL: https://github.com/apache/hadoop/pull/5717#issuecomment-1654687014 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

[GitHub] [hadoop] slfan1989 commented on pull request #5890: YARN-11538. CS UI: queue filter do not work as expected when submitti…

2023-07-27 Thread via GitHub
slfan1989 commented on PR #5890: URL: https://github.com/apache/hadoop/pull/5890#issuecomment-1654639182 We need to Fix CheckStyle issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

[GitHub] [hadoop] hadoop-yetus commented on pull request #5896: YARN-11543: Fix checkstyle issues after YARN-11520.

2023-07-27 Thread via GitHub
hadoop-yetus commented on PR #5896: URL: https://github.com/apache/hadoop/pull/5896#issuecomment-1654467420 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reex

[GitHub] [hadoop] hadoop-yetus commented on pull request #5879: HDFS-17130. Blocks on IN_MAINTENANCE DNs should be sorted properly in LocatedBlocks.

2023-07-27 Thread via GitHub
hadoop-yetus commented on PR #5879: URL: https://github.com/apache/hadoop/pull/5879#issuecomment-1654138046 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

[GitHub] [hadoop] ctubbsii commented on pull request #5789: HADOOP-18786. Use CDN instead of ASF archive

2023-07-27 Thread via GitHub
ctubbsii commented on PR #5789: URL: https://github.com/apache/hadoop/pull/5789#issuecomment-1654127004 @steveloughran wrote: > @ctubbsii do we need to bump up the versions for the move to downloads to work? The old versions only exist in the archives, so yes. But, as previously

[jira] [Commented] (HADOOP-18786) Hadoop build depends on archives.apache.org

2023-07-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17748283#comment-17748283 ] ASF GitHub Bot commented on HADOOP-18786: - ctubbsii commented on PR #5789: URL:

[jira] [Commented] (HADOOP-18829) s3a prefetch LRU cache eviction metric

2023-07-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17748268#comment-17748268 ] ASF GitHub Bot commented on HADOOP-18829: - virajjasani commented on PR #5893: U

[GitHub] [hadoop] virajjasani commented on pull request #5893: HADOOP-18829. S3A prefetch LRU cache eviction metric

2023-07-27 Thread via GitHub
virajjasani commented on PR #5893: URL: https://github.com/apache/hadoop/pull/5893#issuecomment-1654075129 @mehakmeet @mukund-thakur this one is one of the follow-ups from HADOOP-18291, could you please review this PR? -- This is an automated message from the Apache Git Service. To respon

[jira] [Commented] (HADOOP-18747) AWS SDK V2 - sigv2 support

2023-07-27 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17748240#comment-17748240 ] Steve Loughran commented on HADOOP-18747: - * we would have to document the CRT

[GitHub] [hadoop] hadoop-yetus commented on pull request #5890: YARN-11538. CS UI: queue filter do not work as expected when submitti…

2023-07-27 Thread via GitHub
hadoop-yetus commented on PR #5890: URL: https://github.com/apache/hadoop/pull/5890#issuecomment-1653876922 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reex

[GitHub] [hadoop] brumi1024 opened a new pull request, #5896: YARN-11543: Fix checkstyle issues after YARN-11520.

2023-07-27 Thread via GitHub
brumi1024 opened a new pull request, #5896: URL: https://github.com/apache/hadoop/pull/5896 ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests b

[GitHub] [hadoop] brumi1024 merged pull request #5871: YARN-11520. Support capacity vector for AQCv2 dynamic templates.

2023-07-27 Thread via GitHub
brumi1024 merged PR #5871: URL: https://github.com/apache/hadoop/pull/5871 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@had

[GitHub] [hadoop] hadoop-yetus commented on pull request #5895: MAPREDUCE-7446. Fix NegativeArraySizeException in IFile's readRawValue method

2023-07-27 Thread via GitHub
hadoop-yetus commented on PR #5895: URL: https://github.com/apache/hadoop/pull/5895#issuecomment-1653806488 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

[GitHub] [hadoop] umamaheswararao commented on pull request #5885: HDFS-17120. Support snapshot diff based copylisting for flat paths.

2023-07-27 Thread via GitHub
umamaheswararao commented on PR #5885: URL: https://github.com/apache/hadoop/pull/5885#issuecomment-1653794909 I agree, hasPathCapability is a good idea. scheme based dependencies have problems with viewfs like setups. otherwise anyway It may need to depend on resolvePaths to make it work c

[GitHub] [hadoop] hadoop-yetus commented on pull request #5890: YARN-11538. CS UI: queue filter do not work as expected when submitti…

2023-07-27 Thread via GitHub
hadoop-yetus commented on PR #5890: URL: https://github.com/apache/hadoop/pull/5890#issuecomment-1653779405 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

[GitHub] [hadoop] hadoop-yetus commented on pull request #5879: HDFS-17130. Blocks on IN_MAINTENANCE DNs should be sorted properly in LocatedBlocks.

2023-07-27 Thread via GitHub
hadoop-yetus commented on PR #5879: URL: https://github.com/apache/hadoop/pull/5879#issuecomment-1653773483 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reex

[jira] [Updated] (HADOOP-18831) Missing null check when running doRun method

2023-07-27 Thread ConfX (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ConfX updated HADOOP-18831: --- Description: h2. What happened? Got NullPointerException when running {{doRun}} method in {{{}ZKFailoverC

[GitHub] [hadoop] brumi1024 commented on pull request #5895: MAPREDUCE-7446. Fix NegativeArraySizeException in IFile's readRawValue method

2023-07-27 Thread via GitHub
brumi1024 commented on PR #5895: URL: https://github.com/apache/hadoop/pull/5895#issuecomment-1653771314 > `// Read key and value lengths currentKeyLength = WritableUtils.readVInt(dIn); currentValueLength = WritableUtils.readVInt(dIn); bytesRead += WritableUtils.getVIntSize(currentKeyLength

[jira] [Created] (HADOOP-18831) Missing null check when running doRun method

2023-07-27 Thread ConfX (Jira)
ConfX created HADOOP-18831: -- Summary: Missing null check when running doRun method Key: HADOOP-18831 URL: https://issues.apache.org/jira/browse/HADOOP-18831 Project: Hadoop Common Issue Type: Bug

[GitHub] [hadoop] tomicooler commented on a diff in pull request #5895: MAPREDUCE-7446. Fix NegativeArraySizeException in IFile's readRawValue method

2023-07-27 Thread via GitHub
tomicooler commented on code in PR #5895: URL: https://github.com/apache/hadoop/pull/5895#discussion_r1276395283 ## hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFile.java: ## @@ -433,8 +434,11 @@ public boo

[jira] [Resolved] (HADOOP-18743) convert declarations of AWS v1 SDK EnvironmentVariableCredentialsProvider to v2 version

2023-07-27 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18743. - Resolution: Duplicate > convert declarations of AWS v1 SDK EnvironmentVariableCredentia

[GitHub] [hadoop] hadoop-yetus commented on pull request #5876: HDFS-17116. RBF: Update invoke millisecond time as monotonicNow() in RouterSafemodeService

2023-07-27 Thread via GitHub
hadoop-yetus commented on PR #5876: URL: https://github.com/apache/hadoop/pull/5876#issuecomment-1653746939 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

[jira] [Assigned] (HADOOP-18830) S3 Select: deprecate vs cut

2023-07-27 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HADOOP-18830: --- Assignee: Steve Loughran > S3 Select: deprecate vs cut > -

[GitHub] [hadoop] steveloughran commented on pull request #5885: HDFS-17120. Support snapshot diff based copylisting for flat paths.

2023-07-27 Thread via GitHub
steveloughran commented on PR #5885: URL: https://github.com/apache/hadoop/pull/5885#issuecomment-1653673632 the way to be adaptive for FS is to use hasPathCapability() on the FS and so ask it what it does, rather than hard code various filesystem schemas. -- This is an automated message

[GitHub] [hadoop] hfutatzhanghb commented on pull request #5889: HDFS-17123. Sort datanodeStorages when generating StorageBlockReport[] in method BPServiceActor#blockReport for future convenience

2023-07-27 Thread via GitHub
hfutatzhanghb commented on PR #5889: URL: https://github.com/apache/hadoop/pull/5889#issuecomment-1653616192 > Thank you very much for your detailed description of the bug, it made me understand. We need to fix the case where there are two internal blocks of the same block group on the same

[GitHub] [hadoop] LiuGuH closed pull request #5892: HDFS-17129. mis-order of ibr and fbr on datanode

2023-07-27 Thread via GitHub
LiuGuH closed pull request #5892: HDFS-17129. mis-order of ibr and fbr on datanode URL: https://github.com/apache/hadoop/pull/5892 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

[GitHub] [hadoop] brumi1024 commented on pull request #5883: YARN-11521. Parameterize tests runs with legacy/non-legacy queue-mode.

2023-07-27 Thread via GitHub
brumi1024 commented on PR #5883: URL: https://github.com/apache/hadoop/pull/5883#issuecomment-1653613030 Thanks @tomicooler for the patch, merged to trunk. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [hadoop] brumi1024 merged pull request #5883: YARN-11521. Parameterize tests runs with legacy/non-legacy queue-mode.

2023-07-27 Thread via GitHub
brumi1024 merged PR #5883: URL: https://github.com/apache/hadoop/pull/5883 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@had

[GitHub] [hadoop] p-szucs commented on a diff in pull request #5895: MAPREDUCE-7446. Fix NegativeArraySizeException in IFile's readRawValue method

2023-07-27 Thread via GitHub
p-szucs commented on code in PR #5895: URL: https://github.com/apache/hadoop/pull/5895#discussion_r1276266771 ## hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFile.java: ## @@ -433,8 +434,11 @@ public boolea

[GitHub] [hadoop] zhangshuyan0 commented on pull request #5889: HDFS-17123. Sort datanodeStorages when generating StorageBlockReport[] in method BPServiceActor#blockReport for future convenience

2023-07-27 Thread via GitHub
zhangshuyan0 commented on PR #5889: URL: https://github.com/apache/hadoop/pull/5889#issuecomment-1653605923 Thank you very much for your detailed description of the bug, it made me understand. We need to fix the case where there are two internal blocks of the same block group on the same DN

[jira] [Commented] (HADOOP-18758) build/release a shaded version of the AWS SDK

2023-07-27 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17748141#comment-17748141 ] Steve Loughran commented on HADOOP-18758: - the aws sdk is shded, it is just a j

[jira] [Created] (HADOOP-18830) S3 Select: deprecate vs cut

2023-07-27 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-18830: --- Summary: S3 Select: deprecate vs cut Key: HADOOP-18830 URL: https://issues.apache.org/jira/browse/HADOOP-18830 Project: Hadoop Common Issue Type: Sub-t

[jira] [Updated] (HADOOP-18812) list AWS SDK v2 libraries in LICENSE-binary

2023-07-27 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18812: Summary: list AWS SDK v2 libraries in LICENSE-binary (was: list aws SDK v2 libraries in

[jira] [Commented] (HADOOP-18820) AWS SDK v2: make the v1 bridging support optional

2023-07-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17748138#comment-17748138 ] ASF GitHub Bot commented on HADOOP-18820: - steveloughran commented on PR #5872:

[GitHub] [hadoop] tomicooler commented on a diff in pull request #5895: MAPREDUCE-7446. Fix NegativeArraySizeException in IFile's readRawValue method

2023-07-27 Thread via GitHub
tomicooler commented on code in PR #5895: URL: https://github.com/apache/hadoop/pull/5895#discussion_r1276203099 ## hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFile.java: ## @@ -433,8 +434,11 @@ public boo

[GitHub] [hadoop] steveloughran commented on pull request #5872: HADOOP-18820. Cut AWS v1 support

2023-07-27 Thread via GitHub
steveloughran commented on PR #5872: URL: https://github.com/apache/hadoop/pull/5872#issuecomment-1653535263 the current pom changes means that s3 select operations fail ``` [ERROR] testReadLandsatRecordsNoMatch(org.apache.hadoop.fs.s3a.select.ITestS3SelectLandsat) Time elapsed:

[GitHub] [hadoop] tomicooler commented on a diff in pull request #5895: MAPREDUCE-7446. Fix NegativeArraySizeException in IFile's readRawValue method

2023-07-27 Thread via GitHub
tomicooler commented on code in PR #5895: URL: https://github.com/apache/hadoop/pull/5895#discussion_r1276203099 ## hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFile.java: ## @@ -433,8 +434,11 @@ public boo

[GitHub] [hadoop] tomicooler commented on a diff in pull request #5895: MAPREDUCE-7446. Fix NegativeArraySizeException in IFile's readRawValue method

2023-07-27 Thread via GitHub
tomicooler commented on code in PR #5895: URL: https://github.com/apache/hadoop/pull/5895#discussion_r1276203099 ## hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFile.java: ## @@ -433,8 +434,11 @@ public boo

[GitHub] [hadoop] hadoop-yetus commented on pull request #5895: MAPREDUCE-7446. Fix NegativeArraySizeException in IFile's readRawValue method

2023-07-27 Thread via GitHub
hadoop-yetus commented on PR #5895: URL: https://github.com/apache/hadoop/pull/5895#issuecomment-1653475780 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

[GitHub] [hadoop] haiyang1987 commented on pull request #5876: HDFS-17116. RBF: Update invoke millisecond time as monotonicNow() in RouterSafemodeService

2023-07-27 Thread via GitHub
haiyang1987 commented on PR #5876: URL: https://github.com/apache/hadoop/pull/5876#issuecomment-1653459621 Hi Sir @Hexiaoqiao @slfan1989 @hfutatzhanghb Update PR and because update implementation need modify the name of the issue. please help me review this pr again when you have fr

[GitHub] [hadoop] hadoop-yetus commented on pull request #5883: YARN-11521. Parameterize tests runs with legacy/non-legacy queue-mode.

2023-07-27 Thread via GitHub
hadoop-yetus commented on PR #5883: URL: https://github.com/apache/hadoop/pull/5883#issuecomment-1653445096 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reex

[GitHub] [hadoop] hadoop-yetus commented on pull request #5871: YARN-11520. Support capacity vector for AQCv2 dynamic templates.

2023-07-27 Thread via GitHub
hadoop-yetus commented on PR #5871: URL: https://github.com/apache/hadoop/pull/5871#issuecomment-1653430836 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reex

[GitHub] [hadoop] hadoop-yetus commented on pull request #5895: MAPREDUCE-7446. Fix NegativeArraySizeException in IFile's readRawValue method

2023-07-27 Thread via GitHub
hadoop-yetus commented on PR #5895: URL: https://github.com/apache/hadoop/pull/5895#issuecomment-1653420333 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

[GitHub] [hadoop] hadoop-yetus commented on pull request #5871: YARN-11520. Support capacity vector for AQCv2 dynamic templates.

2023-07-27 Thread via GitHub
hadoop-yetus commented on PR #5871: URL: https://github.com/apache/hadoop/pull/5871#issuecomment-1653415340 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reex

[GitHub] [hadoop] hadoop-yetus commented on pull request #5881: Hadoop-18759: [ABFS][Backoff-Optimization] Have a Static retry policy for connection timeout.

2023-07-27 Thread via GitHub
hadoop-yetus commented on PR #5881: URL: https://github.com/apache/hadoop/pull/5881#issuecomment-1653412562 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reex

[GitHub] [hadoop] hadoop-yetus commented on pull request #5879: HDFS-17130. Blocks on IN_MAINTENANCE DNs should be sorted properly in LocatedBlocks.

2023-07-27 Thread via GitHub
hadoop-yetus commented on PR #5879: URL: https://github.com/apache/hadoop/pull/5879#issuecomment-1653411616 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reex

[GitHub] [hadoop] hadoop-yetus commented on pull request #5862: YARN-11536. [Federation] Router CLI Supports Batch Save the SubClusterPolicyConfiguration Of Queues.

2023-07-27 Thread via GitHub
hadoop-yetus commented on PR #5862: URL: https://github.com/apache/hadoop/pull/5862#issuecomment-1653406105 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

[GitHub] [hadoop] hadoop-yetus commented on pull request #5894: YARN-11542. Fix NegativeArraySizeException in IFile's readRawValue me…

2023-07-27 Thread via GitHub
hadoop-yetus commented on PR #5894: URL: https://github.com/apache/hadoop/pull/5894#issuecomment-1653402342 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

[GitHub] [hadoop] ferdelyi commented on pull request #5895: MAPREDUCE-7446. Fix NegativeArraySizeException in IFile's readRawValue method

2023-07-27 Thread via GitHub
ferdelyi commented on PR #5895: URL: https://github.com/apache/hadoop/pull/5895#issuecomment-1653356337 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

[GitHub] [hadoop] hadoop-yetus commented on pull request #5825: HDFS-17072. DFSAdmin: add a triggerDirectoryScanner command.

2023-07-27 Thread via GitHub
hadoop-yetus commented on PR #5825: URL: https://github.com/apache/hadoop/pull/5825#issuecomment-1653338604 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

[GitHub] [hadoop] steveloughran commented on pull request #5789: HADOOP-18786. Use CDN instead of ASF archive

2023-07-27 Thread via GitHub
steveloughran commented on PR #5789: URL: https://github.com/apache/hadoop/pull/5789#issuecomment-1653336705 reviewing this. @ctubbsii do we need to bump up the versions for the move to downloads to work? -- This is an automated message from the Apache Git Service. To respond to

[jira] [Commented] (HADOOP-18786) Hadoop build depends on archives.apache.org

2023-07-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17748069#comment-17748069 ] ASF GitHub Bot commented on HADOOP-18786: - steveloughran commented on PR #5789:

[jira] [Commented] (HADOOP-18819) AWS SDK v2 build complaints

2023-07-27 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17747910#comment-17747910 ] Steve Loughran commented on HADOOP-18819: - do the javadoc bits of this in HADOO

[jira] [Commented] (HADOOP-18812) list aws SDK v2 libraries in LICENSE.binary

2023-07-27 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17747900#comment-17747900 ] Steve Loughran commented on HADOOP-18812: - fixing in HADOOP-18820 > list aws S

[jira] [Commented] (HADOOP-18829) s3a prefetch LRU cache eviction metric

2023-07-27 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17747889#comment-17747889 ] ASF GitHub Bot commented on HADOOP-18829: - hadoop-yetus commented on PR #5893:

[GitHub] [hadoop] hadoop-yetus commented on pull request #5893: HADOOP-18829. S3A prefetch LRU cache eviction metric

2023-07-27 Thread via GitHub
hadoop-yetus commented on PR #5893: URL: https://github.com/apache/hadoop/pull/5893#issuecomment-1653282739 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reex

[GitHub] [hadoop] p-szucs closed pull request #5894: YARN-11542. Fix NegativeArraySizeException in IFile's readRawValue me…

2023-07-27 Thread via GitHub
p-szucs closed pull request #5894: YARN-11542. Fix NegativeArraySizeException in IFile's readRawValue me… URL: https://github.com/apache/hadoop/pull/5894 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [hadoop] p-szucs opened a new pull request, #5895: MAPREDUCE-7446. Fix NegativeArraySizeException in IFile's readRawValue method

2023-07-27 Thread via GitHub
p-szucs opened a new pull request, #5895: URL: https://github.com/apache/hadoop/pull/5895 Change-Id: I0c0226208521976af3acaff92f89c92bd8607921 ### Description of PR We are using bit shifting to double the byte array in IFile's nextRawValue method to store the byte values in

[GitHub] [hadoop] p-szucs opened a new pull request, #5894: YARN-11542. Fix NegativeArraySizeException in IFile's readRawValue me…

2023-07-27 Thread via GitHub
p-szucs opened a new pull request, #5894: URL: https://github.com/apache/hadoop/pull/5894 …thod Change-Id: Ie943e06fe287ed08abd51060bacd58b05a465811 ### Description of PR We are using bit shifting to double the byte array in IFile's nextRawValue method to store the b

[GitHub] [hadoop] hadoop-yetus commented on pull request #5892: HDFS-17129. mis-order of ibr and fbr on datanode

2023-07-27 Thread via GitHub
hadoop-yetus commented on PR #5892: URL: https://github.com/apache/hadoop/pull/5892#issuecomment-1653198151 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

[GitHub] [hadoop] zhiubok commented on pull request #5879: HDFS-17130. Blocks on IN_MAINTENANCE DNs should be sorted properly in LocatedBlocks.

2023-07-27 Thread via GitHub
zhiubok commented on PR #5879: URL: https://github.com/apache/hadoop/pull/5879#issuecomment-1653142901 > In DatanodeManager, there is a comment on line 545 that needs to be modified. > > * For striped blocks, it will only move decommissioned/decommissioning/stale/slow > > shou

[GitHub] [hadoop] hadoop-yetus commented on pull request #5888: HDFS-17121. BPServiceActor to provide new thread to handle FBR

2023-07-27 Thread via GitHub
hadoop-yetus commented on PR #5888: URL: https://github.com/apache/hadoop/pull/5888#issuecomment-1653106180 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

[GitHub] [hadoop] sadanand48 commented on pull request #5885: HDFS-17120. Support snapshot diff based copylisting for flat paths.

2023-07-27 Thread via GitHub
sadanand48 commented on PR #5885: URL: https://github.com/apache/hadoop/pull/5885#issuecomment-1653093330 Filed HDFS-17131 for the same. Thanks @umamaheswararao , @ayushtkn , @swamirishi . -- This is an automated message from the Apache Git Service. To respond to the message, please log o

[GitHub] [hadoop] umamaheswararao commented on pull request #5885: HDFS-17120. Support snapshot diff based copylisting for flat paths.

2023-07-27 Thread via GitHub
umamaheswararao commented on PR #5885: URL: https://github.com/apache/hadoop/pull/5885#issuecomment-1653089202 Thanks @sadanand48 for the contribution. Thanks @swamirishi and @ayushtkn for the reviews. -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [hadoop] umamaheswararao merged pull request #5885: HDFS-17120. Support snapshot diff based copylisting for flat paths.

2023-07-27 Thread via GitHub
umamaheswararao merged PR #5885: URL: https://github.com/apache/hadoop/pull/5885 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr.

[GitHub] [hadoop] umamaheswararao commented on pull request #5885: HDFS-17120. Support snapshot diff based copylisting for flat paths.

2023-07-27 Thread via GitHub
umamaheswararao commented on PR #5885: URL: https://github.com/apache/hadoop/pull/5885#issuecomment-1653086998 @sadanand48 could you please file the followup JIRAs as you thinking? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

[GitHub] [hadoop] brumi1024 merged pull request #5868: YARN-11539. Fix leaf-templates in Flexible AQC.

2023-07-27 Thread via GitHub
brumi1024 merged PR #5868: URL: https://github.com/apache/hadoop/pull/5868 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@had

[GitHub] [hadoop] sadanand48 commented on pull request #5885: HDFS-17120. Support snapshot diff based copylisting for flat paths.

2023-07-27 Thread via GitHub
sadanand48 commented on PR #5885: URL: https://github.com/apache/hadoop/pull/5885#issuecomment-1653039629 Thanks @ayushtkn for taking a look. 1. If the config (`traverseDirectories`) is set to false in case of HDFS, it will cause `DuplicateFileException` and fail the distcp operation.