[PR] Use thread pool to improve the speed of creating control files in TestDFSIO [hadoop]

2023-11-23 Thread via GitHub
hfutatzhanghb opened a new pull request, #6294: URL: https://github.com/apache/hadoop/pull/6294 ### Description of PR When we use TestDFSIO tool to test the throughouts of HDFS clusters, we found it is so slow in the creating controll files stage. After refering to the source

[jira] [Updated] (HADOOP-18989) Use thread pool to improve the speed of creating control files in TestDFSIO

2023-11-23 Thread farmmamba (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] farmmamba updated HADOOP-18989: --- Description: When we use TestDFSIO tool to test the throughouts of HDFS clusters, we found it is

Re: [PR] YARN-11616. Fast fail for NodeConstraintParser when having multi attribute kvs [hadoop]

2023-11-23 Thread via GitHub
zhuqi-lucas merged PR #6282: URL: https://github.com/apache/hadoop/pull/6282 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Created] (HADOOP-18989) Use thread pool to improve the speed of creating control files in TestDFSIO

2023-11-23 Thread farmmamba (Jira)
farmmamba created HADOOP-18989: -- Summary: Use thread pool to improve the speed of creating control files in TestDFSIO Key: HADOOP-18989 URL: https://issues.apache.org/jira/browse/HADOOP-18989 Project:

Re: [PR] HADOOP-18971: [ABFS] Enable Footer Read Optimizations with Appropriate Footer Read Buffer Size [hadoop]

2023-11-23 Thread via GitHub
anujmodi2021 commented on PR #6270: URL: https://github.com/apache/hadoop/pull/6270#issuecomment-1825197578 @steveloughran @mukund-thakur @mehakmeet Gentle reminder for reviewing this PR. Thanks -- This is an automated message from the Apache Git Service. To respond to the message,

[jira] [Commented] (HADOOP-18971) ABFS: Enable Footer Read Optimizations with Appropriate Footer Read Buffer Size

2023-11-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17789338#comment-17789338 ] ASF GitHub Bot commented on HADOOP-18971: - anujmodi2021 commented on code in PR #6270: URL:

Re: [PR] HADOOP-18971: [ABFS] Enable Footer Read Optimizations with Appropriate Footer Read Buffer Size [hadoop]

2023-11-23 Thread via GitHub
anujmodi2021 commented on code in PR #6270: URL: https://github.com/apache/hadoop/pull/6270#discussion_r1403976331 ## hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsInputStreamReadFooter.java: ## @@ -387,8 +374,72 @@ private void

[jira] [Commented] (HADOOP-18910) ABFS: Adding Support for MD5 Hash based integrity verification of the request content during transport

2023-11-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17789337#comment-17789337 ] ASF GitHub Bot commented on HADOOP-18910: - anujmodi2021 commented on PR #6069: URL:

Re: [PR] HADOOP-18910: [ABFS] Adding Support for MD5 Hash based integrity verification of the request content during transport [hadoop]

2023-11-23 Thread via GitHub
anujmodi2021 commented on PR #6069: URL: https://github.com/apache/hadoop/pull/6069#issuecomment-1825191803 @steveloughran, @mukund-thakur @mehakmeet Gentle Reminder to review this PR. All comments addressed. Thanks -- This is an automated message from the Apache Git Service. To

Re: [PR] Hadoop-18759: [ABFS][Backoff-Optimization] Have a Static retry policy for connection timeout. [hadoop]

2023-11-23 Thread via GitHub
anujmodi2021 commented on PR #5881: URL: https://github.com/apache/hadoop/pull/5881#issuecomment-1825187277 @mukund-thakur @mehakmeet @steveloughran Gentle reminder for this PR. This has been active for a while. Steve has also kindly reviewed the PR with his approval. Thanks --

[jira] [Commented] (HADOOP-18915) S3A client builder to use getTImeDuration() for HTTP timeouts

2023-11-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17789282#comment-17789282 ] ASF GitHub Bot commented on HADOOP-18915: - hadoop-yetus commented on PR #6180: URL:

[jira] [Updated] (HADOOP-18915) S3A client builder to use getTImeDuration() for HTTP timeouts

2023-11-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-18915: Labels: pull-request-available (was: ) > S3A client builder to use getTImeDuration()

Re: [PR] HADOOP-18915. use getDuration() on http connection settings [hadoop]

2023-11-23 Thread via GitHub
hadoop-yetus commented on PR #6180: URL: https://github.com/apache/hadoop/pull/6180#issuecomment-1825044265 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

Re: [PR] Hadoop 18860: Upgrade mockito version to 4.11.0 [hadoop]

2023-11-23 Thread via GitHub
hadoop-yetus commented on PR #6275: URL: https://github.com/apache/hadoop/pull/6275#issuecomment-1824847734 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18708) AWS SDK V2 - Implement CSE

2023-11-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17789242#comment-17789242 ] ASF GitHub Bot commented on HADOOP-18708: - hadoop-yetus commented on PR #6164: URL:

Re: [PR] HADOOP-18708. AWS SDK V2 - Implement CSE [hadoop]

2023-11-23 Thread via GitHub
hadoop-yetus commented on PR #6164: URL: https://github.com/apache/hadoop/pull/6164#issuecomment-1824846036 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18708) AWS SDK V2 - Implement CSE

2023-11-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17789241#comment-17789241 ] ASF GitHub Bot commented on HADOOP-18708: - hadoop-yetus commented on PR #6164: URL:

Re: [PR] HADOOP-18708. AWS SDK V2 - Implement CSE [hadoop]

2023-11-23 Thread via GitHub
hadoop-yetus commented on PR #6164: URL: https://github.com/apache/hadoop/pull/6164#issuecomment-1824843569 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18708) AWS SDK V2 - Implement CSE

2023-11-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17789240#comment-17789240 ] ASF GitHub Bot commented on HADOOP-18708: - hadoop-yetus commented on PR #6164: URL:

Re: [PR] HADOOP-18708. AWS SDK V2 - Implement CSE [hadoop]

2023-11-23 Thread via GitHub
hadoop-yetus commented on PR #6164: URL: https://github.com/apache/hadoop/pull/6164#issuecomment-1824841327 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18708) AWS SDK V2 - Implement CSE

2023-11-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17789238#comment-17789238 ] ASF GitHub Bot commented on HADOOP-18708: - hadoop-yetus commented on PR #6164: URL:

Re: [PR] HADOOP-18708. AWS SDK V2 - Implement CSE [hadoop]

2023-11-23 Thread via GitHub
hadoop-yetus commented on PR #6164: URL: https://github.com/apache/hadoop/pull/6164#issuecomment-1824840177 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18708) AWS SDK V2 - Implement CSE

2023-11-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17789219#comment-17789219 ] ASF GitHub Bot commented on HADOOP-18708: - ahmarsuhail commented on PR #6164: URL:

Re: [PR] HADOOP-18708. AWS SDK V2 - Implement CSE [hadoop]

2023-11-23 Thread via GitHub
ahmarsuhail commented on PR #6164: URL: https://github.com/apache/hadoop/pull/6164#issuecomment-1824745814 thanks @steveloughran , addressed review comments -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] HADOOP-18708. AWS SDK V2 - Implement CSE [hadoop]

2023-11-23 Thread via GitHub
hadoop-yetus commented on PR #6164: URL: https://github.com/apache/hadoop/pull/6164#issuecomment-1824622944 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Commented] (HADOOP-18708) AWS SDK V2 - Implement CSE

2023-11-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17789179#comment-17789179 ] ASF GitHub Bot commented on HADOOP-18708: - hadoop-yetus commented on PR #6164: URL:

Re: [PR] YARN-11420 Stabilize TestNMClient [hadoop]

2023-11-23 Thread via GitHub
hadoop-yetus commented on PR #5317: URL: https://github.com/apache/hadoop/pull/5317#issuecomment-1824408034 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: |

[jira] [Assigned] (HADOOP-18987) Corrections to Hadoop FileSystem API Definition

2023-11-23 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HADOOP-18987: --- Assignee: Dieter De Paepe > Corrections to Hadoop FileSystem API Definition >

[jira] [Commented] (HADOOP-18987) Corrections to Hadoop FileSystem API Definition

2023-11-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17789115#comment-17789115 ] ASF GitHub Bot commented on HADOOP-18987: - steveloughran commented on code in PR #6292: URL:

Re: [PR] HADOOP-18987 Various fixes to FileSystem API docs [hadoop]

2023-11-23 Thread via GitHub
steveloughran commented on code in PR #6292: URL: https://github.com/apache/hadoop/pull/6292#discussion_r1403319403 ## hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md: ## @@ -432,7 +432,7 @@ of data which must be collected in a single RPC call.

[jira] [Updated] (HADOOP-18987) Corrections to Hadoop FileSystem API Definition

2023-11-23 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18987: Affects Version/s: 3.3.6 > Corrections to Hadoop FileSystem API Definition >

[jira] [Created] (HADOOP-18988) Snapshotdiff does not detect rename if parent dir removed

2023-11-23 Thread Dieter De Paepe (Jira)
Dieter De Paepe created HADOOP-18988: Summary: Snapshotdiff does not detect rename if parent dir removed Key: HADOOP-18988 URL: https://issues.apache.org/jira/browse/HADOOP-18988 Project: Hadoop

[jira] [Commented] (HADOOP-18959) Use builder for prefetch CachingBlockManager

2023-11-23 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17789064#comment-17789064 ] ASF GitHub Bot commented on HADOOP-18959: - ahmarsuhail merged PR #6240: URL:

Re: [PR] HADOOP-18959 S3A : Use builder for prefetch CachingBlockManager [hadoop]

2023-11-23 Thread via GitHub
ahmarsuhail merged PR #6240: URL: https://github.com/apache/hadoop/pull/6240 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] YARN-11420 Stabilize TestNMClient [hadoop]

2023-11-23 Thread via GitHub
susheel-gupta commented on code in PR #5317: URL: https://github.com/apache/hadoop/pull/5317#discussion_r1403154800 ## hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java: ## @@ -456,6

Re: [PR] HDFS-17223. Add journalnode maintenance node list [hadoop]

2023-11-23 Thread via GitHub
gp1314 commented on PR #6183: URL: https://github.com/apache/hadoop/pull/6183#issuecomment-1824082862 @Hexiaoqiao , could you help to review this improvement in the code? If it's feasible, can it be merged into the trunk branch? I would be very grateful. -- This is an automated message